Validio supports integrating with dbt (data build tool) to validate metadata for test results and model runs. To integrate with dbt Core,

1. Add a warehouse credential for the warehouse where your dbt output is stored.
2. Add a dbt Core credential which will be used when uploading your dbt files.
3. Upload your dbt manifest and run results files to Validio.
4. Add dbt sources for monitoring model runs and test results.

Prerequisites for Integrating with dbt Core

You must have the Validio CLI or Validio SDK installed on the machine or image that runs dbt. You can use either service to upload your dbt manifest and run results files to Validio. (Installing the CLI will automatically install the Validio SDK.) For more information, see Getting Started with Developer Toolkit and Validio CLI.

Add a Warehouse Credential

Before creating a dbt Core credential, you must create a valid credential for the warehouse where your dbt output is stored. You will reference this warehouse credential when you configure the dbt Core credential. For specific credential setup instructions, see Data Warehouse Sources and select the relevant warehouse (for example, BigQuery or PostgreSQL).

Add a dbt Core Credential

To add a credential for monitoring dbt sources,

1. Navigate to Observability > Credentials and click + Add Credential.
2. Under Credential type, select dbt Core Credential.
3. Fill in the configuration parameters. For more information, see dbt Core Credential Parameters.
4. Click Create.

Validio will perform a validation step. Unlike with the dbt Cloud Credential, Validio cannot automatically fetch the data from your dbt Core account. Instead, you will need to manually upload your dbt files for model runs and test results to Validio.

dbt Core Credential Parameters

Upload dbt Files to Validio

You can use either the Validio CLI or Validio SDK to upload your dbt manifest and run files to Validio.

Upload with Validio CLI

In the Validio CLI, run the validio dbt upload command as shown in the following example. For available options, refer to the dbt CLI Parameters table.

DBT_CREDENTIAL_ID="my-dbtCore-credential_12345"

validio dbt upload --target-path ./target --job-name local-test --credential-id $DBT_CREDENTIAL_ID

Upload with Validio SDK

The following is an example of how to use the Validio SDK to upload the dbt files to Validio. For available options, refer to the dbt SDK Parameters table.

from validio_sdk.validio_client import ValidioAPIClient
from validio_sdk.graphql_client.input_types import DbtArtifactUploadInput

from pathlib import Path

client = ValidioAPIClient()
client.dbt_artifact_upload(DbtArtifactUploadInput(
  credentialId='',
  jobName='',
  manifest=Path("/tmp/target/manifest.json").read_text*(),

Add a dbt Source

Validio provides two types of dbt sources that you can add for monitoring dbt runs and tests: dbt Model Runs and dbt Test Results.

To add a dbt source:

1. Navigate to Observability > Sources and click + New Source.
2. Under Source type, select either dbt Model Runs or dbt Test Results.
3. Under Credentials,
   1. Select Use existing Credential.
   2. Select the valid credential from the list.
4. Under Config,
   1. Enter a Name for the source.
   2. Select a Project Name and Job Name.
   3. Select a Cron preset to specify the polling frequency for the data.
5. Under Window, select the Window type, which should be either Tumbling Window or Global Window. For more information, see Configuring Windows.
6. Click Continue to create the source.

You can view the summary of the source configuration.

📘

Note

Validio automatically infers the schema and defines default segmentation.

• For dbt Model Runs, the default segment is model_id.
• For dbt Test Results, the default segments are model_id and test_name.