Using Catalog

Data Catalog automatically discovers assets based on the Credentials you have added to Validio and enabled for catalog. For more information, see About Credentials.

📘

Note

It may take longer for the catalog assets to populate the first time that a catalog job runs for a new credential.

The Catalog page organizes your data assets into tabs based on what they are: tables, streams, object storage services, business intelligence tools, and activities. From the Tables, Streams, Object Storage and BI Tools tabs, you can do the following:

• Search and filter to quickly find relevant assets
• View more information about an asset
• Add custom descriptions to the asset
• Convert an asset to a source
• Add or edit tags to an asset
• View relationships with other assets in Lineage

Search and Filter Assets

If you have a large number of catalog assets, you can search and filter to show only relevant assets in the list. For example, you can search for specific assets using keywords and then apply quick filters to narrow your results further by selected values. You can search and filter in all the catalog assets tabs except the Activities tab.

• Catalog Search–Type keywords into the search bar and the asset list will update automatically to show matching results, highlighting the keywords that match in the Name and Description fields. When you type more than one keyword, the results will return all assets that include any of the keywords.
• Quick Filters–Selecting one or more values of Type, Credential, or Label to quickly filter the list of assets. Clicking on the filters also shows a count of the assets that match each value.

📘

Note

The results of searching and filtering will respect the sorting that has been applied to the table.

Refresh Catalog Assets

You can manually refresh your catalog to discover new assets since the last refresh. When refreshing your catalog assets, you can select specific credentials or refresh all assets.

• You can only choose credentials that have been enabled for catalog.
• A full refresh of assets performs over the past 30 days.

View Asset Details

You can view more information about the asset or source by clicking on the row in the table to open the details panel or clicking the name of the asset to open its Source details page.

• The details panel includes descriptions and metadata that is indexed and searchable. The panel also displays utilization statistics as well as configuration and ownership information. From this panel, you can also add tags, convert the asset to a source, or view the asset or source in the Lineage page.
• If you click on the asset name to open its Source details page, you will only have access to the Overview and Schema tags. You can also convert the asset to a source from this page. For more information, see About Source Details.

Add Custom Catalog Descriptions

Validio imports descriptions and tags automatically from external systems, such as BigQuery and dbt. You can also add your own descriptions to a catalog asset. All existing descriptions are included in the Description column of the assets table, with icons to indicate where they originate.

To add a custom catalog description,

1. Click the asset Name in the table to open its source details page.
2. In the Overview tab, click the edit icon next to Add description.
3. Type in your description.
4. Click the check mark to save your description.

When you return to the Catalog page, the custom description will display in the table with the Validio icon next to it.

Convert an Asset to a Source

Converting an asset to a source allows you to validate the data and the schema of the asset. You can convert any asset in the catalog to a source. For more information, see About Sources.

To convert an asset to a source,

1. Click the asset row or asset name in the table.
2. In the details panel or source details for the asset, click Convert to Source to open the configuration wizard.
3. Review the information for the Catalog asset and click Continue.
4. Under Config,
   1. Enter a Name for the new source.
   2. (Optional) Select the Cron preset and, if needed, edit the Cron expression to define the polling frequency.
5. Under Schema,
   1. Select all the Fields that you want to validate for the new source.
   2. Select a Cursor field, which represents a timestamp when data is updated.
   3. Select a Lookback time, which indicates how far back in time to start reading data from the source.
6. Under Window,
   1. Select a Window type. For Data Warehouse sources, Validio recommends Tumbling windows or Global windows. For more information, see Data Warehouse Sources.
   2. Configure the window based on your selected window type. For more information, see About Windows.
7. Click Continue to create the source.

After the source is created, you can create a segmentation, go to the source details page, or go back to sources.

Add Asset Tags

You can add tags organize and prioritize your data assets with descriptive labels or other relevant information. Use the details panel to add tags to an asset or source.

View Asset in Lineage

From the details panel for an asset, click View Lineage to open the Lineage page where you can view the relationships with other assets. For more information, see About Lineage.