This guide is part of the Getting Started Tutorial. For more information, see the Tutorial Overview.

Your demo account is configured with the Validio GCP Demo Credential. This demo credential gives access to our Consumer Tech datasets. You can view a list of the assets associated with the Consumer Tech credential in the Catalog page.

In this guide, you will set up sources to monitor the Consumer Tech data, add a validator to each source, and start the source with backfill to include historical data in your monitoring and validation.

Add a Source to Monitor

Although you can view the assets associated with a credential, Validio does not monitor data until you connect it to a data source. An easy way to start monitoring data is by converting a catalog asset into a source:

1. Navigate to Catalog > Table assets.
   The list displays all the table assets you have access to with the demo credential.
2. Find and check the row for the gold_daily_management_data asset.
3. Click Convert to Source to open the configuration wizard.
4. Under Catalog asset,
   1. Credential: select the Validio GCP Demo Credential to associate with the asset when converting it to a source.
   2. Set how many days of Historic Data to use when you start the source.
   3. Specify how frequently the validators on the source will poll or check for changes. With the default configuration, Validio polls for new data from this table every 6 hours.
   4. Click Continue. For BigQuery assets, the next step will be adding recommended validators to monitor metadata on the sources.
5. Under Validators, select the metadata validators to add to your source.
6. Click Continue to create the gold_daily_management_data source.
7. After the source is created, click View Source to open the source details page on the Validators tab.

Add Validators to the Source

In the previous section, you converted the table gold_daily_management_data from an asset into a source. Before you start the source, you will add validators which define the data and metrics you want to monitor and the thresholds to detect incidents. In this section, you will add two types of validators:

• A Freshness validator to monitor the pipeline health of your source. Freshness validators check if any new data points have been added when Validio polls the source for updates. If there is no new data, there may be an issue with the pipeline. For more information, see Freshness Validator.
• A Numeric validator to monitor a metric calculation for the sum of the field nr_of_crashes_daily. The numeric validator calculates the sum value over a daily window, and then validates if these values follow an expected pattern. For more information, see Numeric Validator.

Add a Freshness Validator

To add a Freshness validator:

1. Navigate to the gold_daily_management_data source page and Validators tab.
2. Click + New validator.
3. Under Validator type, select Pipeline Health > Freshness.
4. Under Metrics,
   1. Keep the default daily window on the timestamp field event_date.
   2. Select Initialize using historic data. The amount of historical data included in this backfill is defined by the Lookback time preset on the source.
5. Under Threshold, click Continue to use Dynamic Thresholds with the default sensitivity.
6. Under Validator details, enter a display name, description, tags, and assign an owner.
7. Click Continue to create the Freshness validator.

You can click View Validator to see the validator's details page. The details page will not be active until you start the source to begin monitoring and validating its data. Before starting the source, let's add the second validator.

Add a Numeric Validator

To add a Numeric validator:

1. Navigate to the gold_daily_management_data source page and Validators tab.
2. Click + New validator.
3. Under Validator type, select Metrics & Validity > Numeric Statistics.
4. Under Metrics,
   1. Select Sum for the Metric calculation to perform and select the numeric field, nr_of_crashes_daily.
   2. Keep the default daily window on the timestamp field event_date.
   3. Select the Segmentation field country in the list.
   4. Select Initialize using historic data. The amount of historical data included in this backfill is defined by the Lookback time preset on the source.
5. Under Threshold, click Continue to use Dynamic Thresholds with the default sensitivity.
6. Under Validator details, enter a display name, description, tags, and assign an owner.
7. Click Continue to create the Sum of "nr_of_crashes_daily" validator.

You can click View Validator to see the validator's details page. The details page will not be active until you start the source to begin monitoring and validating its data.

Start the Source with Backfill

Monitoring and validation doesn't start until you run the source and its validators.

1. Navigate to the gold_daily_management_data source page and Validators tab.
2. Click Start to run the source and its validators.
   Give the source a few minutes to run. Because you configured the source with a lookback period of 1 month and you configured the validators to start with backfill, you will see historic data populate the incidents graph. Including historical data allows Validio to learn and improve its anomaly detection on future data.

Add More Sources and Validators

Now, convert two more catalog assets into sources and add a validator to each one:

1. Add the source bronze_mobile_app_events with the Volume validator Count on "is_event". Configure the validator with segmentation on the fields country and device_model. All other settings can be default.
2. Add the source silver_crash_daily_statistics with the Numeric validator Sum of "nr_of_crashes". Configure the validator with no segmentation and all other default settings.
3. Start the sources. After a few minutes, you will see more data populate the incidents graphs.