Using Lineage

The Lineage page displays a knowledge graph that allows you to track and visualize how data flows through the pipelines of your data stack. The lineage graph presents data flow from left to right, with upstream entities to the left and downstream entities to the right.

Add and Change Anchor

You can add an entry point or anchor to your global lineage graph. This anchor defines the specific catalog asset and field to focus on when you navigate to the Lineage page.

To add an anchor,

1. Select catalog asset to anchor lineage to.
2. (Optional) Select a specific field to anchor lineage to.
3. Click Set as anchor.

You also have the option to change the anchor using the Change anchor button.

Add a Metric to the Graph

You can select a metric to display the Number of Reads, Number of Writes, or Utilization over the catalog assets in the graph.

Navigate the Lineage Graph

To navigate the lineage graph,

• Use the Zoom In, Zoom Out, and Reset buttons.
• Click and drag the cursor to zoom into a selection within the graph.
• Hold the shift key and drag the cursor to pan or move around within the graph.

View Lineage Details

You can view more information about the catalog assets and relationships in a details panel.

To view more information about an asset or source, click on the dataset in the graph. The details panel includes indexed descriptions and metadata that is searchable. The panel also displays utilization statistics as well as configuration and ownership information. From this panel, you can add tags, convert the asset to a source, or view the asset or source in its Source details page.

To view more information about relationships, click on an edge or connection in the graph. The details panel includes the upstream and downstream assets and the SQL query that defines the relationship.

Convert an Asset to a Source

Converting an asset to a source allows you to validate the data and the schema of the asset. You can convert any asset in the lineage graph to a source. For more information, see About Sources.

To convert an asset into a source,

1. Click on the asset in the graph.
2. In the details panel, click Convert to Source to open the configuration wizard.
3. Review the information for the Catalog asset and click Continue.
4. Under Config, enter a Name for the new source.
5. Under Schema,
   1. Select all the Fields that you want to validate for the new source.
   2. Select a Cursor field, which represents a timestamp when data is updated.
   3. Select a Lookback time, which indicates how far back in time to start reading data from the source.
6. Under Window,
   1. Select a Window type. For Data Warehouse sources, Validio recommends Tumbling windows or Global windows. For more information, see Data Warehouse Sources.
   2. Configure the window based on your selected window type. For more information, see About Windows.
7. Click Continue to create the source.

Connect Assets

To manually add lineage relations to the graph,

1. Click Connect Assets.
2. Enter or select the Upstream asset, which is the dataset the edge starts from.
3. (Optional) Select the upstream Field, which is the specific field where the edge starts. If no field is selected, the edge will start from the dataset.
4. Enter or select the Downstream asset, which is the dataset the edge ends at.
5. (Optional) Select the downstream Field, which is the specific field where the edge ends. If no field is selected, the edge will end at the dataset.
6. Click Connect Assets.

Delete Connections

To delete a connection,

1. Click a connection in the graph.
2. In the details panel for the Connection, click Delete edge.
3. Click Confirm.