Introduction to Cloudera Navigator Lineage Diagrams

Required Role:

A lineage diagram is a directed graph that depicts an extracted entity and its relations with other entities. A lineage diagram is limited to 3000 entities.

There are two types of lineage diagrams:
  • Template - represents an entity that is a model for other entities
  • Instance - represents an instance or execution of a template

Entities

In a lineage diagram, entity types are represented by icons:
HDFS Pig
  • File
  • Directory
  • Table
  • Pig script
  • Pig script execution
Hive and Impala Spark (Unsupported and disabled by default. To enable, see Enabling Spark Metadata Extraction.)
  • Table
  • Query template
  • Query execution
  • Job template
  • Job execution
MapReduce and YARN Sqoop
  • Job template
  • Job execution
  • Job template
  • Job execution
Oozie    
  • Job template
  • Job execution
   
In the following circumstances the entity type icon will appear as :
  • The entity has not yet been extracted. In this case will eventually be replaced with the correct entity icon after the entity is extracted and linked in Navigator. For information on how long it takes for newly created entities to be extracted, see Metadata Extraction.
  • A Hive entity has been deleted from the system before it could be extracted.

Parent entities are represented by a white box enclosing other entities. The following lineage diagram illustrates the relations between the YARN job script.pig and Pig script script.pig invoked by the parent Oozie workflow pig-app-hue-script and its source file midsummer.txt and destination folder upperout:


Relations

Relations between the entities are represented graphically by gray lines, with arrows indicating the direction of the data flow. There are the following types of relations:

Relation Type Description
Data flow Describes a relation between data and a processing activity. For example, between a file and a MapReduce job or vice versa.
Alias Describes an alias relation. For example, from a table to a synonym.
Parent-child Describes a parent-child relation. For example, between a directory and a file.
Logical-physical Describes the relation between a logical entity and its physical entity. For example, between a Hive query and a MapReduce job.
Conjoint Describes a non-directional relation. For example, between a table and an index.
Instance of Describes the relation between a template and its instance. For example, an operation execution is an instance of operation.
Control flow Describes a relation where the source entity controls the data flow of the target entity. For example, between the columns used in an insert clause and the where clause of a Hive query.
Lineage diagrams display the following line types:
  • A solid line () represents a "data flow" relationship, indicating that the columns will appear (possibly transformed) in the output. For example, a solid line will appear between the columns used in a select clause.
  • A dashed line () represents a "control flow" relationship, indicating that the columns determine which rows will flow to the output. For example, a dashed line will appear between the columns used in an insert or select clause and the where clause of a Hive query.
The following query:
SELECT sample_07.description,sample_07.salary FROM sample_07
WHERE ( sample_07.salary > 100000)
ORDER BY sample_07.salary DESC LIMIT 1000
has solid lines between the columns in the select clause and a dashed line between the columns in the where clause:

Manipulating Lineage Diagrams

You can click a parent entity to display its child entities. For example, you can click the Pig script to display its child tables:


  • To improve the layout of a lineage diagram you can drag and drop entities (in this case midsummer.txt and upperout) located outside a parent box.
  • You can use the mouse scroll wheel to zoom the lineage diagram in and out.
  • You can move the lineage diagram in the lineage pane by pressing the mouse button and dragging it.

Displaying a Template Lineage Diagram

A template lineage diagram contains template entities, such as jobs and queries, that can be instantiated, and the input and output entities to which they are related.

To display a template lineage diagram:
  1. Perform a metadata search.
  2. In the list of results, click an Operation or Query result entry. For example, when you click the sample_09 result entry:

    the Search screen is replaced with a page that displays the entity property sheet on the left and lineage diagram on the right:

The selected entity sample_09 appears with a white box as a background.
This example lineage diagram illustrates the relations between a Hive query execution entity and its source and destination tables:

When you click each entity icon, columns and lines connecting the source and destination columns display:

If you hover over a part, the source and destination columns are highlighted:

Displaying an Instance Lineage Diagram

An instance lineage diagram displays instance entities, such as job and query executions, and the input and output entities to which they are related.

To display an instance lineage diagram:
  1. Display a template lineage diagram. For example:

  2. Click the Instances tab, which contains a list of links to instances of the template.
  3. Click a link to display an instance lineage diagram. For the preceding template diagram, the job instance job_1426651548889_0004 replaces the word count job template.

Displaying the Template Lineage Diagram for an Instance Lineage Diagram

You can navigate from an instance diagram to its template.

  1. Display an instance lineage diagram.
  2. Click the value of the Template property to go to the instance's template.

Downloading a Lineage File

Lineage is externalized in a lineage file in JSON format.

  1. Display a template or instance lineage diagram.
  2. Click the icon at the top left of the diagram.
    A lineage file named lineage.json is downloaded. For example, the lineage file representing job_1426651548889_0004 from the preceding section is: Tracing through the relations shows that job_1426651548889_0004, which has the identity 69b79a8c0c7701f316dd86894b97fe58, has the INSTANCE_OF relation with word count and the DATA_FLOW relation with /user/hdfs/input and /user/hdfs/out1.
    {
      "entities": {
        "01043ab3a019a68f37f3d33efa122f0f": {
          "level": 1,
          "physical": [],
          "logical": [],
          "aliasOf": [],
          "aliases": [],
          "instances": [],
          "children": [],
          "workflows": [],
          "identity": "01043ab3a019a68f37f3d33efa122f0f",
          "originalName": "part-r-00001",
          "sourceId": "a09b0233cc58ff7d601eaa68673a20c6",
          "firstClassParentId": null,
          "parentPath": "/user/hdfs/out1",
          "extractorRunId": "a09b0233cc58ff7d601eaa68673a20c6##1372",
          "name": "part-r-00001",
          "description": null,
          "tags": null,
          "fileSystemPath": "/user/hdfs/out1/part-r-00001",
          "type": "FILE",
          "size": 8,
          "created": "2015-03-27T17:44:20.639Z",
          "lastModified": "2015-03-27T17:44:20.639Z",
          "lastAccessed": "2015-03-27T17:44:16.832Z",
          "permissions": "rw-r--r--",
          "owner": "hdfs",
          "group": "supergroup",
          "blockSize": null,
          "mimeType": "application/octet-stream",
          "replication": null,
          "userEntity": false,
          "deleted": false,
          "sourceType": "HDFS",
          "internalType": "fselement",
          "nameField": "originalName",
          "sourceName": "HDFS-1",
          "hueLink": "http://tcdn2-1.ent.cloudera.com:8888/filebrowser/view/user/hdfs/out1/part-r-00001",
          "isScript": false,
          "hasUpstream": true,
          "parent": "89612c409b76f7bdf00036df9c3cb717",
          "activeChildren": []
        },
        "72c31f8dbe14a520bd46a747d1382d89": {
          "level": 1,
          "physical": [],
          "logical": [],
          "aliasOf": [],
          "aliases": [],
          "instances": [],
          "children": [
            "f2eca1680ecca38fa514dc191613c7b4",
            "f3929c0b9b2a16490ee57e0a498eee5e"
          ],
          "workflows": [],
          "identity": "72c31f8dbe14a520bd46a747d1382d89",
          "originalName": "input",
          "sourceId": "a09b0233cc58ff7d601eaa68673a20c6",
          "firstClassParentId": null,
          "parentPath": "/user/hdfs",
          "extractorRunId": "a09b0233cc58ff7d601eaa68673a20c6##1370",
          "name": "input",
          "description": null,
          "tags": null,
          "fileSystemPath": "/user/hdfs/input",
          "type": "DIRECTORY",
          "size": null,
          "created": "2015-03-27T17:40:43.665Z",
          "lastModified": "2015-03-27T17:41:06.825Z",
          "lastAccessed": null,
          "permissions": "rwxr-xr-x",
          "owner": "hdfs",
          "group": "supergroup",
          "blockSize": null,
          "mimeType": null,
          "replication": null,
          "userEntity": false,
          "deleted": false,
          "sourceType": "HDFS",
          "internalType": "fselement",
          "nameField": "originalName",
          "sourceName": "HDFS-1",
          "hueLink": "http://tcdn2-1.ent.cloudera.com:8888/filebrowser/view/user/hdfs/input",
          "isScript": false,
          "hasDownstream": true,
          "column": -1,
          "renderOrdinal": 1,
          "activeChildren": [
            {
              "level": 1,
              "physical": [],
              "logical": [],
              "aliasOf": [],
              "aliases": [],
              "instances": [],
              "children": [],
              "workflows": [],
              "identity": "f3929c0b9b2a16490ee57e0a498eee5e",
              "originalName": "test.txt",
              "sourceId": "a09b0233cc58ff7d601eaa68673a20c6",
              "firstClassParentId": null,
              "parentPath": "/user/hdfs/input",
              "extractorRunId": "a09b0233cc58ff7d601eaa68673a20c6##1370",
              "name": "test.txt",
              "description": null,
              "tags": null,
              "fileSystemPath": "/user/hdfs/input/test.txt",
              "type": "FILE",
              "size": 6,
              "created": "2015-03-27T17:41:06.825Z",
              "lastModified": "2015-03-27T17:41:06.825Z",
              "lastAccessed": "2015-03-27T17:41:06.405Z",
              "permissions": "rw-r--r--",
              "owner": "hdfs",
              "group": "supergroup",
              "blockSize": null,
              "mimeType": "application/octet-stream",
              "replication": null,
              "userEntity": false,
              "deleted": false,
              "sourceType": "HDFS",
              "internalType": "fselement",
              "nameField": "originalName",
              "sourceName": "HDFS-1",
              "hueLink": "http://tcdn2-1.ent.cloudera.com:8888/filebrowser/view/user/hdfs/input/test.txt",
              "isScript": false,
              "hasDownstream": true,
              "parent": "72c31f8dbe14a520bd46a747d1382d89",
              "activeChildren": []
            }
          ],
          "x": -222.4375,
          "y": -52
        },
        "f2eca1680ecca38fa514dc191613c7b4": {
          "level": 1,
          "physical": [],
          "logical": [],
          "aliasOf": [],
          "aliases": [],
          "instances": [],
          "children": [],
          "workflows": [],
          "identity": "f2eca1680ecca38fa514dc191613c7b4",
          "originalName": "test.txt._COPYING_",
          "sourceId": "a09b0233cc58ff7d601eaa68673a20c6",
          "firstClassParentId": null,
          "parentPath": "/user/hdfs/input",
          "extractorRunId": "a09b0233cc58ff7d601eaa68673a20c6##1370",
          "name": "test.txt._COPYING_",
          "description": null,
          "tags": null,
          "fileSystemPath": "/user/hdfs/input/test.txt._COPYING_",
          "type": "FILE",
          "size": 6,
          "created": "2015-03-27T17:41:06.405Z",
          "lastModified": "2015-03-27T17:41:06.405Z",
          "lastAccessed": "2015-03-27T17:41:06.405Z",
          "permissions": "rw-r--r--",
          "owner": "hdfs",
          "group": "supergroup",
          "blockSize": null,
          "mimeType": "application/octet-stream",
          "replication": null,
          "userEntity": false,
          "deleted": true,
          "sourceType": "HDFS",
          "internalType": "fselement",
          "nameField": "originalName",
          "sourceName": "HDFS-1",
          "hueLink": "http://tcdn2-1.ent.cloudera.com:8888/filebrowser/view/user/hdfs/input/test.txt._COPYING_",
          "isScript": false,
          "parent": "72c31f8dbe14a520bd46a747d1382d89",
          "activeChildren": []
        },
        "16b093b257033463bab26bba4c707450": {
          "level": 1,
          "physical": [],
          "logical": [],
          "aliasOf": [],
          "aliases": [],
          "instances": [],
          "children": [],
          "workflows": [],
          "identity": "16b093b257033463bab26bba4c707450",
          "originalName": "_temporary",
          "sourceId": "a09b0233cc58ff7d601eaa68673a20c6",
          "firstClassParentId": null,
          "parentPath": "/user/hdfs/out1",
          "extractorRunId": "a09b0233cc58ff7d601eaa68673a20c6##1372",
          "name": "_temporary",
          "description": null,
          "tags": null,
          "fileSystemPath": "/user/hdfs/out1/_temporary",
          "type": "DIRECTORY",
          "size": null,
          "created": "2015-03-27T17:41:32.486Z",
          "lastModified": "2015-03-27T17:41:32.486Z",
          "lastAccessed": null,
          "permissions": "rwxr-xr-x",
          "owner": "hdfs",
          "group": "supergroup",
          "blockSize": null,
          "mimeType": null,
          "replication": null,
          "userEntity": false,
          "deleted": false,
          "sourceType": "HDFS",
          "internalType": "fselement",
          "nameField": "originalName",
          "sourceName": "HDFS-1",
          "hueLink": "http://tcdn2-1.ent.cloudera.com:8888/filebrowser/view/user/hdfs/out1/_temporary",
          "isScript": false,
          "parent": "89612c409b76f7bdf00036df9c3cb717",
          "activeChildren": []
        },
        "89612c409b76f7bdf00036df9c3cb717": {
          "level": 1,
          "physical": [],
          "logical": [],
          "aliasOf": [],
          "aliases": [],
          "instances": [],
          "children": [
            "fcd80476d5a968e29e86411b4a67af87",
            "01043ab3a019a68f37f3d33efa122f0f",
            "16b093b257033463bab26bba4c707450",
            "75470b40586cde9e092a01d37798d921"
          ],
          "workflows": [],
          "identity": "89612c409b76f7bdf00036df9c3cb717",
          "originalName": "out1",
          "sourceId": "a09b0233cc58ff7d601eaa68673a20c6",
          "firstClassParentId": null,
          "parentPath": "/user/hdfs",
          "extractorRunId": "a09b0233cc58ff7d601eaa68673a20c6##1372",
          "name": "out1",
          "description": null,
          "tags": null,
          "fileSystemPath": "/user/hdfs/out1",
          "type": "DIRECTORY",
          "size": null,
          "created": "2015-03-27T17:41:32.486Z",
          "lastModified": "2015-03-27T17:44:20.848Z",
          "lastAccessed": null,
          "permissions": "rwxr-xr-x",
          "owner": "hdfs",
          "group": "supergroup",
          "blockSize": null,
          "mimeType": null,
          "replication": null,
          "userEntity": false,
          "deleted": false,
          "sourceType": "HDFS",
          "internalType": "fselement",
          "nameField": "originalName",
          "sourceName": "HDFS-1",
          "hueLink": "http://tcdn2-1.ent.cloudera.com:8888/filebrowser/view/user/hdfs/out1",
          "isScript": false,
          "hasUpstream": true,
          "column": 1,
          "renderOrdinal": 2,
          "activeChildren": [
            {
              "level": 1,
              "physical": [],
              "logical": [],
              "aliasOf": [],
              "aliases": [],
              "instances": [],
              "children": [],
              "workflows": [],
              "identity": "fcd80476d5a968e29e86411b4a67af87",
              "originalName": "_SUCCESS",
              "sourceId": "a09b0233cc58ff7d601eaa68673a20c6",
              "firstClassParentId": null,
              "parentPath": "/user/hdfs/out1",
              "extractorRunId": "a09b0233cc58ff7d601eaa68673a20c6##1372",
              "name": "_SUCCESS",
              "description": null,
              "tags": null,
              "fileSystemPath": "/user/hdfs/out1/_SUCCESS",
              "type": "FILE",
              "size": 0,
              "created": "2015-03-27T17:44:20.848Z",
              "lastModified": "2015-03-27T17:44:20.848Z",
              "lastAccessed": "2015-03-27T17:44:20.848Z",
              "permissions": "rw-r--r--",
              "owner": "hdfs",
              "group": "supergroup",
              "blockSize": null,
              "mimeType": "application/octet-stream",
              "replication": null,
              "userEntity": false,
              "deleted": false,
              "sourceType": "HDFS",
              "internalType": "fselement",
              "nameField": "originalName",
              "sourceName": "HDFS-1",
              "hueLink": "http://tcdn2-1.ent.cloudera.com:8888/filebrowser/view/user/hdfs/out1/_SUCCESS",
              "isScript": false,
              "parent": "89612c409b76f7bdf00036df9c3cb717",
              "hasUpstream": true,
              "activeChildren": []
            },
            {
              "level": 1,
              "physical": [],
              "logical": [],
              "aliasOf": [],
              "aliases": [],
              "instances": [],
              "children": [],
              "workflows": [],
              "identity": "75470b40586cde9e092a01d37798d921",
              "originalName": "part-r-00000",
              "sourceId": "a09b0233cc58ff7d601eaa68673a20c6",
              "firstClassParentId": null,
              "parentPath": "/user/hdfs/out1",
              "extractorRunId": "a09b0233cc58ff7d601eaa68673a20c6##1372",
              "name": "part-r-00000",
              "description": null,
              "tags": null,
              "fileSystemPath": "/user/hdfs/out1/part-r-00000",
              "type": "FILE",
              "size": 0,
              "created": "2015-03-27T17:44:20.576Z",
              "lastModified": "2015-03-27T17:44:20.576Z",
              "lastAccessed": "2015-03-27T17:44:16.831Z",
              "permissions": "rw-r--r--",
              "owner": "hdfs",
              "group": "supergroup",
              "blockSize": null,
              "mimeType": "application/octet-stream",
              "replication": null,
              "userEntity": false,
              "deleted": false,
              "sourceType": "HDFS",
              "internalType": "fselement",
              "nameField": "originalName",
              "sourceName": "HDFS-1",
              "hueLink": "http://tcdn2-1.ent.cloudera.com:8888/filebrowser/view/user/hdfs/out1/part-r-00000",
              "isScript": false,
              "hasUpstream": true,
              "parent": "89612c409b76f7bdf00036df9c3cb717",
              "activeChildren": []
            },
            {
              "level": 1,
              "physical": [],
              "logical": [],
              "aliasOf": [],
              "aliases": [],
              "instances": [],
              "children": [],
              "workflows": [],
              "identity": "01043ab3a019a68f37f3d33efa122f0f",
              "originalName": "part-r-00001",
              "sourceId": "a09b0233cc58ff7d601eaa68673a20c6",
              "firstClassParentId": null,
              "parentPath": "/user/hdfs/out1",
              "extractorRunId": "a09b0233cc58ff7d601eaa68673a20c6##1372",
              "name": "part-r-00001",
              "description": null,
              "tags": null,
              "fileSystemPath": "/user/hdfs/out1/part-r-00001",
              "type": "FILE",
              "size": 8,
              "created": "2015-03-27T17:44:20.639Z",
              "lastModified": "2015-03-27T17:44:20.639Z",
              "lastAccessed": "2015-03-27T17:44:16.832Z",
              "permissions": "rw-r--r--",
              "owner": "hdfs",
              "group": "supergroup",
              "blockSize": null,
              "mimeType": "application/octet-stream",
              "replication": null,
              "userEntity": false,
              "deleted": false,
              "sourceType": "HDFS",
              "internalType": "fselement",
              "nameField": "originalName",
              "sourceName": "HDFS-1",
              "hueLink": "http://tcdn2-1.ent.cloudera.com:8888/filebrowser/view/user/hdfs/out1/part-r-00001",
              "isScript": false,
              "hasUpstream": true,
              "parent": "89612c409b76f7bdf00036df9c3cb717",
              "activeChildren": []
            }
          ],
          "x": 222.4375,
          "y": -52
        },
        "a3ac8013155effa2f96e9de0f177eeb5": {
          "level": 1,
          "physical": [],
          "logical": [],
          "aliasOf": [],
          "aliases": [],
          "instances": [
            "69b79a8c0c7701f316dd86894b97fe58"
          ],
          "children": [],
          "workflows": [],
          "identity": "a3ac8013155effa2f96e9de0f177eeb5",
          "originalName": "word count",
          "sourceId": "a063e69e6c0660353dc378c836837935",
          "firstClassParentId": null,
          "parentPath": null,
          "extractorRunId": "a063e69e6c0660353dc378c836837935##1381",
          "name": "word count",
          "description": null,
          "tags": null,
          "wfIds": null,
          "inputFormat": null,
          "outputFormat": null,
          "outputKey": "org.apache.hadoop.io.Text",
          "outputValue": "org.apache.hadoop.io.IntWritable",
          "mapper": "org.apache.hadoop.examples.WordCount$TokenizerMapper",
          "reducer": "org.apache.hadoop.examples.WordCount$IntSumReducer",
          "sourceType": "YARN",
          "type": "OPERATION",
          "userEntity": false,
          "deleted": null,
          "internalType": "mrjobspec",
          "nameField": "name",
          "sourceName": "YARN-1",
          "isScript": false
        },
        "69b79a8c0c7701f316dd86894b97fe58": {
          "level": 1,
          "physical": [],
          "logical": [],
          "aliasOf": [],
          "aliases": [],
          "instances": [],
          "children": [],
          "workflows": [],
          "identity": "69b79a8c0c7701f316dd86894b97fe58",
          "originalName": "job_1426651548889_0004",
          "sourceId": "a063e69e6c0660353dc378c836837935",
          "firstClassParentId": null,
          "parentPath": null,
          "extractorRunId": "a063e69e6c0660353dc378c836837935##1381",
          "name": "job_1426651548889_0004",
          "description": null,
          "tags": null,
          "started": "2015-03-27T17:41:20.896Z",
          "ended": "2015-03-27T17:44:21.969Z",
          "principal": "hdfs",
          "inputs": [
            "hdfs://tcdn2-1.ent.cloudera.com:8020/user/hdfs/input"
          ],
          "outputs": [
            "hdfs://tcdn2-1.ent.cloudera.com:8020/user/hdfs/out1"
          ],
          "wfInstId": null,
          "jobID": "job_1426651548889_0004",
          "sourceType": "YARN",
          "inputRecursive": false,
          "type": "OPERATION_EXECUTION",
          "userEntity": false,
          "deleted": null,
          "internalType": "mrjobinstance",
          "nameField": "originalName",
          "sourceName": "YARN-1",
          "hueLink": "http://tcdn2-1.ent.cloudera.com:8888/jobbrowser/jobs/application_1426651548889_0004",
          "isScript": false,
          "hasDownstream": true,
          "hasUpstream": true,
          "template": "a3ac8013155effa2f96e9de0f177eeb5",
          "active": true,
          "column": 0,
          "renderOrdinal": 0,
          "activeChildren": [],
          "x": 0,
          "y": -52
        },
        "75470b40586cde9e092a01d37798d921": {
          "level": 1,
          "physical": [],
          "logical": [],
          "aliasOf": [],
          "aliases": [],
          "instances": [],
          "children": [],
          "workflows": [],
          "identity": "75470b40586cde9e092a01d37798d921",
          "originalName": "part-r-00000",
          "sourceId": "a09b0233cc58ff7d601eaa68673a20c6",
          "firstClassParentId": null,
          "parentPath": "/user/hdfs/out1",
          "extractorRunId": "a09b0233cc58ff7d601eaa68673a20c6##1372",
          "name": "part-r-00000",
          "description": null,
          "tags": null,
          "fileSystemPath": "/user/hdfs/out1/part-r-00000",
          "type": "FILE",
          "size": 0,
          "created": "2015-03-27T17:44:20.576Z",
          "lastModified": "2015-03-27T17:44:20.576Z",
          "lastAccessed": "2015-03-27T17:44:16.831Z",
          "permissions": "rw-r--r--",
          "owner": "hdfs",
          "group": "supergroup",
          "blockSize": null,
          "mimeType": "application/octet-stream",
          "replication": null,
          "userEntity": false,
          "deleted": false,
          "sourceType": "HDFS",
          "internalType": "fselement",
          "nameField": "originalName",
          "sourceName": "HDFS-1",
          "hueLink": "http://tcdn2-1.ent.cloudera.com:8888/filebrowser/view/user/hdfs/out1/part-r-00000",
          "isScript": false,
          "hasUpstream": true,
          "parent": "89612c409b76f7bdf00036df9c3cb717",
          "activeChildren": []
        },
        "fcd80476d5a968e29e86411b4a67af87": {
          "level": 1,
          "physical": [],
          "logical": [],
          "aliasOf": [],
          "aliases": [],
          "instances": [],
          "children": [],
          "workflows": [],
          "identity": "fcd80476d5a968e29e86411b4a67af87",
          "originalName": "_SUCCESS",
          "sourceId": "a09b0233cc58ff7d601eaa68673a20c6",
          "firstClassParentId": null,
          "parentPath": "/user/hdfs/out1",
          "extractorRunId": "a09b0233cc58ff7d601eaa68673a20c6##1372",
          "name": "_SUCCESS",
          "description": null,
          "tags": null,
          "fileSystemPath": "/user/hdfs/out1/_SUCCESS",
          "type": "FILE",
          "size": 0,
          "created": "2015-03-27T17:44:20.848Z",
          "lastModified": "2015-03-27T17:44:20.848Z",
          "lastAccessed": "2015-03-27T17:44:20.848Z",
          "permissions": "rw-r--r--",
          "owner": "hdfs",
          "group": "supergroup",
          "blockSize": null,
          "mimeType": "application/octet-stream",
          "replication": null,
          "userEntity": false,
          "deleted": false,
          "sourceType": "HDFS",
          "internalType": "fselement",
          "nameField": "originalName",
          "sourceName": "HDFS-1",
          "hueLink": "http://tcdn2-1.ent.cloudera.com:8888/filebrowser/view/user/hdfs/out1/_SUCCESS",
          "isScript": false,
          "parent": "89612c409b76f7bdf00036df9c3cb717",
          "hasUpstream": true,
          "activeChildren": []
        },
        "f3929c0b9b2a16490ee57e0a498eee5e": {
          "level": 1,
          "physical": [],
          "logical": [],
          "aliasOf": [],
          "aliases": [],
          "instances": [],
          "children": [],
          "workflows": [],
          "identity": "f3929c0b9b2a16490ee57e0a498eee5e",
          "originalName": "test.txt",
          "sourceId": "a09b0233cc58ff7d601eaa68673a20c6",
          "firstClassParentId": null,
          "parentPath": "/user/hdfs/input",
          "extractorRunId": "a09b0233cc58ff7d601eaa68673a20c6##1370",
          "name": "test.txt",
          "description": null,
          "tags": null,
          "fileSystemPath": "/user/hdfs/input/test.txt",
          "type": "FILE",
          "size": 6,
          "created": "2015-03-27T17:41:06.825Z",
          "lastModified": "2015-03-27T17:41:06.825Z",
          "lastAccessed": "2015-03-27T17:41:06.405Z",
          "permissions": "rw-r--r--",
          "owner": "hdfs",
          "group": "supergroup",
          "blockSize": null,
          "mimeType": "application/octet-stream",
          "replication": null,
          "userEntity": false,
          "deleted": false,
          "sourceType": "HDFS",
          "internalType": "fselement",
          "nameField": "originalName",
          "sourceName": "HDFS-1",
          "hueLink": "http://tcdn2-1.ent.cloudera.com:8888/filebrowser/view/user/hdfs/input/test.txt",
          "isScript": false,
          "hasDownstream": true,
          "parent": "72c31f8dbe14a520bd46a747d1382d89",
          "activeChildren": []
        }
      },
      "relations": {
        "bd3fe737364968a8fbc1831fc9915dca": {
          "identity": "bd3fe737364968a8fbc1831fc9915dca",
          "type": "DATA_FLOW",
          "propagatorId": "268fc2fbba566558b83abd0f0fb680a1",
          "extractorRunId": "a09b0233cc58ff7d601eaa68673a20c6##1372",
          "sources": {
            "entityIds": [
              "69b79a8c0c7701f316dd86894b97fe58"
            ]
          },
          "targets": {
            "entityIds": [
              "01043ab3a019a68f37f3d33efa122f0f",
              "75470b40586cde9e092a01d37798d921"
            ]
          },
          "propagatable": false,
          "unlinked": false,
          "userSpecified": false
        },
        "33535116782b0baff207851f9e637cf2": {
          "identity": "33535116782b0baff207851f9e637cf2",
          "type": "DATA_FLOW",
          "propagatorId": "217788ca1d4de53a4071cf026299744f",
          "extractorRunId": "a09b0233cc58ff7d601eaa68673a20c6##1372",
          "sources": {
            "entityIds": [
              "f3929c0b9b2a16490ee57e0a498eee5e"
            ]
          },
          "targets": {
            "entityIds": [
              "69b79a8c0c7701f316dd86894b97fe58"
            ]
          },
          "propagatable": false,
          "unlinked": false,
          "userSpecified": false
        },
        "646e2547f1f1371e99259069f3bbd4db": {
          "identity": "646e2547f1f1371e99259069f3bbd4db",
          "type": "PARENT_CHILD",
          "propagatorId": null,
          "extractorRunId": "a09b0233cc58ff7d601eaa68673a20c6##1370",
          "children": {
            "entityIds": [
              "f2eca1680ecca38fa514dc191613c7b4"
            ]
          },
          "parent": {
            "entityId": "72c31f8dbe14a520bd46a747d1382d89"
          },
          "propagatable": false,
          "unlinked": false,
          "userSpecified": false
        },
        "da3e6b9ccbc9e39de59e85ea6d89fdd7": {
          "identity": "da3e6b9ccbc9e39de59e85ea6d89fdd7",
          "type": "PARENT_CHILD",
          "propagatorId": null,
          "extractorRunId": "a09b0233cc58ff7d601eaa68673a20c6##1372",
          "children": {
            "entityIds": [
              "fcd80476d5a968e29e86411b4a67af87"
            ]
          },
          "parent": {
            "entityId": "89612c409b76f7bdf00036df9c3cb717"
          },
          "propagatable": false,
          "unlinked": false,
          "userSpecified": false
        },
        "15816a23933df14590026425fc0e8d85": {
          "identity": "15816a23933df14590026425fc0e8d85",
          "type": "PARENT_CHILD",
          "propagatorId": null,
          "extractorRunId": "a09b0233cc58ff7d601eaa68673a20c6##1372",
          "children": {
            "entityIds": [
              "01043ab3a019a68f37f3d33efa122f0f"
            ]
          },
          "parent": {
            "entityId": "89612c409b76f7bdf00036df9c3cb717"
          },
          "propagatable": false,
          "unlinked": false,
          "userSpecified": false
        },
        "f8f31d2c2638c22f17600a32631c5639": {
          "identity": "f8f31d2c2638c22f17600a32631c5639",
          "type": "PARENT_CHILD",
          "propagatorId": null,
          "extractorRunId": "a09b0233cc58ff7d601eaa68673a20c6##1370",
          "children": {
            "entityIds": [
              "16b093b257033463bab26bba4c707450"
            ]
          },
          "parent": {
            "entityId": "89612c409b76f7bdf00036df9c3cb717"
          },
          "propagatable": false,
          "unlinked": false,
          "userSpecified": false
        },
        "3dcd15d16d13786480052adbac5e7f7f": {
          "identity": "3dcd15d16d13786480052adbac5e7f7f",
          "type": "DATA_FLOW",
          "propagatorId": "268fc2fbba566558b83abd0f0fb680a1",
          "extractorRunId": "a09b0233cc58ff7d601eaa68673a20c6##1372",
          "sources": {
            "entityIds": [
              "69b79a8c0c7701f316dd86894b97fe58"
            ]
          },
          "targets": {
            "entityIds": [
              "75470b40586cde9e092a01d37798d921",
              "01043ab3a019a68f37f3d33efa122f0f",
              "fcd80476d5a968e29e86411b4a67af87"
            ]
          },
          "propagatable": false,
          "unlinked": false,
          "userSpecified": false
        },
        "268fc2fbba566558b83abd0f0fb680a1": {
          "identity": "268fc2fbba566558b83abd0f0fb680a1",
          "type": "DATA_FLOW",
          "propagatorId": null,
          "extractorRunId": "a09b0233cc58ff7d601eaa68673a20c6##1372",
          "sources": {
            "entityIds": [
              "69b79a8c0c7701f316dd86894b97fe58"
            ]
          },
          "targets": {
            "entityIds": [
              "89612c409b76f7bdf00036df9c3cb717"
            ]
          },
          "propagatable": false,
          "unlinked": false,
          "userSpecified": false
        },
        "dd299c827ecde0a1c721b396903cc97d": {
          "identity": "dd299c827ecde0a1c721b396903cc97d",
          "type": "PARENT_CHILD",
          "propagatorId": null,
          "extractorRunId": "a09b0233cc58ff7d601eaa68673a20c6##1370",
          "children": {
            "entityIds": [
              "f3929c0b9b2a16490ee57e0a498eee5e"
            ]
          },
          "parent": {
            "entityId": "72c31f8dbe14a520bd46a747d1382d89"
          },
          "propagatable": false,
          "unlinked": false,
          "userSpecified": false
        },
        "217788ca1d4de53a4071cf026299744f": {
          "identity": "217788ca1d4de53a4071cf026299744f",
          "type": "DATA_FLOW",
          "propagatorId": null,
          "extractorRunId": "a09b0233cc58ff7d601eaa68673a20c6##1372",
          "sources": {
            "entityIds": [
              "72c31f8dbe14a520bd46a747d1382d89"
            ]
          },
          "targets": {
            "entityIds": [
              "69b79a8c0c7701f316dd86894b97fe58"
            ]
          },
          "propagatable": false,
          "unlinked": false,
          "userSpecified": false
        },
        "e0680ada742c6fa1ad3a6192bc2a9274": {
          "identity": "e0680ada742c6fa1ad3a6192bc2a9274",
          "type": "PARENT_CHILD",
          "propagatorId": null,
          "extractorRunId": "a09b0233cc58ff7d601eaa68673a20c6##1372",
          "children": {
            "entityIds": [
              "75470b40586cde9e092a01d37798d921"
            ]
          },
          "parent": {
            "entityId": "89612c409b76f7bdf00036df9c3cb717"
          },
          "propagatable": false,
          "unlinked": false,
          "userSpecified": false
        },
        "234107762623c89b811fd0be8a96676a": {
          "identity": "234107762623c89b811fd0be8a96676a",
          "type": "INSTANCE_OF",
          "propagatorId": null,
          "extractorRunId": "a063e69e6c0660353dc378c836837935##1381",
          "instances": {
            "entityIds": [
              "69b79a8c0c7701f316dd86894b97fe58"
            ]
          },
          "template": {
            "entityId": "a3ac8013155effa2f96e9de0f177eeb5"
          },
          "propagatable": false,
          "unlinked": false,
          "userSpecified": false
        },
        "38358d90c0c9675c76626148732a63a4": {
          "identity": "38358d90c0c9675c76626148732a63a4",
          "type": "DATA_FLOW",
          "propagatorId": "268fc2fbba566558b83abd0f0fb680a1",
          "extractorRunId": "a09b0233cc58ff7d601eaa68673a20c6##1372",
          "sources": {
            "entityIds": [
              "69b79a8c0c7701f316dd86894b97fe58"
            ]
          },
          "targets": {
            "entityIds": [
              "01043ab3a019a68f37f3d33efa122f0f",
              "75470b40586cde9e092a01d37798d921"
            ]
          },
          "propagatable": false,
          "unlinked": false,
          "userSpecified": false
        }
      }
    }