The dbt Core integration connects Validio to your local dbt projects for data transformation monitoring. Validio validates metadata for test results and model runs. To integrate with dbt Core,

1. Add a warehouse credential for the warehouse where your dbt output is stored.
2. Add a dbt Core credential which will be used when uploading your dbt files.
3. Upload your dbt manifest and run results files to Validio.
4. Add dbt sources for monitoring model runs and test results.

Validio also supports integrating with the managed service version of dbt. For more information, see dbt Cloud.

Prerequisites for Integrating with dbt Core

You must have the Validio CLI or Validio SDK installed on the machine or image that runs dbt. You can use either service to upload your dbt manifest and run results files to Validio. (Installing the CLI will automatically install the Validio SDK.) For more information, see Getting Started with Developer Toolkit and Validio CLI.

Add a Warehouse Credential

Before creating a dbt Core credential, you must create a valid credential for the warehouse where your dbt output is stored. You will reference this warehouse credential when you configure the dbt Core credential. For specific credential setup instructions, see Data Warehouse Sources and select the relevant warehouse (for example, BigQuery or PostgreSQL).

Add a dbt Core Credential

To add a credential for monitoring dbt sources,

1. Navigate to Credentials and click + New Credential.
2. Under Namespace, select a namespace where the resources will be created.
3. For Credential Type, select dbt Core.
4. Fill in the Configuration parameter fields. Refer to the dbt Core Credential Parameters table.
5. (Optional) Click Test credential to validate the connection. If validation fails, check that you provided the correct parameter values.
6. Click Create credential.

Unlike with the dbt Cloud credential, Validio cannot automatically fetch the data from your dbt Core account. Instead, you must manually upload your dbt files for model runs and test results to Validio.

dbt Core Credential Parameters

Upload dbt Files to Validio

You can use either the Validio CLI or Validio SDK to upload your dbt manifest and run files to Validio.

Upload with Validio CLI

In the Validio CLI, run the validio dbt upload command as shown in the following example. For available options, refer to the dbt CLI Parameters table.

DBT_CREDENTIAL_ID="my-dbtCore-credential_12345"

validio dbt upload --target-path ./target --job-name local-test --credential-id $DBT_CREDENTIAL_ID

Upload with Validio SDK

The following is an example of how to use the Validio SDK to upload the dbt files to Validio. For available options, refer to the dbt SDK Parameters table.

import asyncio
from pathlib import Path

from validio_sdk import dbt, util
from validio_sdk.client import Client


async def main():
    client = Client()
    await dbt.upload_artifacts(
        client=client,
        credential_id="my-dbtCore-credential_12345",
        job_name="local-test",
        manifest=util.read_json_file(Path("./target/manifest.json")),
        run_results=util.read_json_file(Path("./target/run_results.json")),
    )


asyncio.run(main())

Add a dbt Source

Validio provides two types of dbt sources that you can add for monitoring dbt runs and tests: dbt Model Run and dbt Test Result.

To add a dbt source:

1. Navigate to Sources and click + New source.
2. Under Source type, click Data transformation to select dbt Model Run or dbt Test Result.
3. Under Config,
   1. Select the valid Credential, or create a new credential to authenticate your connection.
   2. Select a Project name and Job name. These correspond to the data you uploaded with validio dbt upload (the --job-name you specified).
   3. Set the Polling schedule (cron expression in UTC), which is how frequently Validio checks for new dbt metadata.
4. Under Source details,
   1. Enter a Name for the source.
   2. (Optional) Add a Description.
   3. Add Tags to help group related sources or to use for routing notifications.
5. Click Continue to create the source.

You can view the summary of the source configuration.