This guide is part of the Getting Started Tutorial. For more information, see the Tutorial Overview.

Your demo account is configured with the Validio GCP Demo Credential. This demo credential gives access to our Consumer Tech datasets. You can view a list of the assets associated with the Consumer Tech credential in the Catalog page.

In this guide, you will set up sources to monitor the Consumer Tech data, add a validator to each source, and start the source with backfill to include historical data in your monitoring and validation.

Add a Source to Monitor

Although you can view the assets associated with a credential, Validio does not monitor data until you connect it to a data source. An easy way to start monitoring data is by converting a catalog asset into a source:

1. Navigate to Catalog > Table assets.
   The list displays all the Table assets you have access to with the demo credential.
2. Find and click gold_daily_management_data.
   The details page for the table asset opens. This page is not active because you are not monitoring the data from this table yet.
3. To monitor gold_daily_management_data, click Convert to Source.
4. Under Catalog asset,
   1. Namespace: Keep the auto-generated namespace.
   2. Credential: select the Validio GCP Demo Credential.
5. Under Config, click Continue to keep the default settings.
   With the default configuration, Validio polls for new data from this table every 6 hours.
6. Under Schema,
   1. Fields: Check Select all to include all automatically detected fields in your schema.
   2. Lookback time preset: Select 1 month.
      The lookback time indicates the amount of historical data to include when you start validating data from the source. Using historical data helps to train Validio's machine learning models. As Validio learns the trends in your data, it automatically sets thresholds to identify anomalies during validation. Note: To include historical data, you will also need to select the Initialize with backfill option when you configure the validator.
7. Under Window, keep the default settings.
   With the default configuration, Validio uses a daily tumbling window to validate your data.
8. Click Continue to create the gold_daily_management_data source.
9. After the source is created, click View Source to open the source details page on the Validators tab.

Add Validators to the Source

In the previous section, you converted the table gold_daily_management_data from an asset into a source. Before you start the source, you will add validators which define the data and metrics you want to monitor and the thresholds to detect incidents. In this section, you will add two types of validators:

• A Freshness validator to monitor the pipeline health of your source. Freshness validators check if any new data points have been added when Validio polls the source for updates. If there is no new data, there may be an issue with the pipeline. For more information, see Freshness Validator.
• A Numeric validator to monitor a metric calculation for the sum of the field nr_of_crashes_daily. The numeric validator calculates the sum value over a daily window, and then validates if these values follow an expected pattern. For more information, see Numeric Validator.

Add a Freshness Validator

To add a Freshness validator:

1. Navigate to the gold_daily_management_data source page and Validators tab.
2. Click + New validator.
3. Under Validator type, select Pipeline Health > Freshness.
4. Under Metrics,
   1. Keep the default daily window on the timestamp field event_date.
   2. Select Initialize using historic data. The amount of historical data included in this backfill is defined by the Lookback time preset on the source.
5. Under Threshold, click Continue to use Dynamic Thresholds with the default sensitivity.
6. Under Validator details, enter a display name, description, tags, and assign an owner.
7. Click Continue to create the Freshness validator.

You can click View Validator to see the validator's details page. The details page will not be active until you start the source to begin monitoring and validating its data. Before starting the source, let's add the second validator.

Add a Numeric Validator

To add a Numeric validator:

1. Navigate to the gold_daily_management_data source page and Validators tab.
2. Click + New validator.
3. Under Validator type, select Metrics & Validity > Numeric Statistics.
4. Under Metrics,
   1. Select Sum for the Metric calculation to perform and select the numeric field, nr_of_crashes_daily.
   2. Keep the default daily window on the timestamp field event_date.
   3. Select the Segmentation field country in the list.
   4. Select Initialize using historic data. The amount of historical data included in this backfill is defined by the Lookback time preset on the source.
5. Under Threshold, click Continue to use Dynamic Thresholds with the default sensitivity.
6. Under Validator details, enter a display name, description, tags, and assign an owner.
7. Click Continue to create the Sum of "nr_of_crashes_daily" validator.

You can click View Validator to see the validator's details page. The details page will not be active until you start the source to begin monitoring and validating its data.

Start the Source with Backfill

Monitoring and validation doesn't start until you run the source and its validators.

1. Navigate to the gold_daily_management_data source page and Validators tab.
2. Click Start to run the source and its validators.
   Give the source a few minutes to run. Because you configured the source with a lookback period of 1 month and you configured the validators to start with backfill, you will see historic data populate the incidents graph. Including historical data allows Validio to learn and improve its anomaly detection on future data.

Add More Sources and Validators

Now, convert two more catalog assets into sources and add a validator to each one:

1. Add the source bronze_mobile_app_events with the Volume validator Count on "is_event". Configure the validator with segmentation on the fields country and device_model. All other settings can be default.
2. Add the source silver_crash_daily_statistics with the Numeric validator Sum of "nr_of_crashes". Configure the validator with no segmentation and all other default settings.
3. Start the sources. After a few minutes, you will see more data populate the incidents graphs.