Validio Code Scenarios

The following sections provide examples for different scenarios using Validio Code:

Update resources: Update the configuration of a resource.
Manage source schema: Fetch upstream schema changes or manually override the schema.
Validate a set of fields: Apply the same validation to a set of fields, or declare filters within a validator.
Destroy resources: Delete all resources in the project’s namespace.
Import resources: Generate code for resources that are not present in the project.

Update resources

Resources are the configured components in your Validio environment, and include channels, credentials, sources, validators, windows, and so on. You can update the configuration of a resource by editing the resource’s declaration in the code. The Write step of the Validio Code workflow includes a main.py example with a 1-hour tumbling window. In the following example, you edit the window definition to disable the window timeout:

Python

window = TumblingWindow(
    name='example-window',
    source=big_query_source,
    data_time_field='event_time',
    window_size=1,
    time_unit=WindowTimeUnit.HOUR,
    # Updated disabling window timeout from False to True
    window_timeout_disabled=True
)

Then, invoke the plan command to display the pending update:

Bash

validio code plan

TumblingWindow 'example-window' will be updated
~ TumblingWindow = {
~   window_timeout_disabled = False => True
~ }


Plan: 0 to create, 1 to update, 0 to delete.

Immutable fields

Some fields are immutable because changing them would drastically change the behavior of the resource and its monitoring results. These fields are not exposed in the user interface, and changing them requires you to remove the resource and set up a new resource with the new settings. For example, the window on a validator is immutable because changing the window would invalidate all historical metrics for that validator. With IaC, changing an immutable field in the manifest automatically removes the old resource and creates the new one for you. If the resource to be recreated has child resources, such as when you change an immutable field on a source, all resources with that source as a parent (such as validators, windows, and segmentations) are also deleted and recreated.

ImportantWhen a source or validator is deleted, all historically processed metrics are also deleted. Once the new configuration is created, you need to run a backfill to recover the historical data.

In IaC, you can review the summary of resources being removed or marked as replaced. The following example shows the output when you change the immutable time_unit field on a tumbling window that is used by two validators:

Bash

validio code plan

TumblingWindow 'tumbling_window' will be replaced: immutable field 'time_unit' was updated
~ TumblingWindow(
~   time_unit = WindowTimeUnit.DAY => WindowTimeUnit.HOUR
~ )

NumericValidator 'mean_of_age' will be replaced: cascading replacement from TumblingWindow(name="tumbling_window")
- NumericValidator(
-   name = mean_of_age
-   window_name = tumbling_window
-   ...
- )

NumericValidator 'mean_of_working_hours_weekly' will be replaced: cascading replacement from TumblingWindow(name="tumbling_window")
- NumericValidator(
-   name = mean_of_working_hours_weekly
-   window_name = tumbling_window
-   ...
- )


Plan: 3 to create, 0 to update, 3 to delete.

All three resources are marked “will be replaced”, and the plan summary shows “3 to create” and “3 to delete”, even though the diff shows an update for the window and only deletes for the validators.

Manage source schema

A source’s configuration includes a set of expected fields and data types, which are defined by the schema. Usually you do not interact with or modify the schema, because Validio automatically detects or infers the schema for all source types. However, you can intervene to manually update the schema to do the following:

Capture upstream schema changes
Override the inferred schema with a modified version

Upstream schema changes

When the schema of the upstream data source changes, for example because of changes in the warehouse table, stream messages, or object storage file, it might add new fields that you would like to monitor. Pass the --update-schema flag to the plan or apply command to fetch the latest schema.

You can pass the --update-schema flag multiple times to check several sources for updates, or use the --update-all-schemas flag to check for schema updates on all sources.

The following example checks for upstream schema updates for the source and reports any changes:

Bash

validio code plan --diff=changes --update-schema example-source

No changes. Configuration is up-to-date!

Manual schema override

You can manually override the inferred schema with a modified version, with the following steps:

Download the inferred schema.
Edit the downloaded schema file.
Pass the contents of the modified schema file as a parameter to the source.

The plan and apply commands detect a change to the source’s schema whenever the contents of the file change. For more information about interpreting and modifying the schema file, see the JSON Type Definition documentation. The following example shows how to manually override an inferred schema with a modified schema file. First, download the inferred schema:

Bash

validio sources infer-schema bigquery \
  --credential-id example-credential \
  --project example-project \
  --dataset example-dataset \
  --table orders \
  --filename example-source-schema.json

Then, edit the downloaded file and pass the contents of the modified schema as a parameter to the source:

Python

from pathlib import Path
from validio_sdk import load_jtd_schema

# Use a manually provided schema instead for the source.
big_query_source = GcpBigQuerySource(
   name="example-source",
   credential=gcp_credential,
   project='example-project',
   dataset='example-dataset',
   table='orders',
   cursor_field='created_at',
   lookback_days=30,
   schedule='0 */12 * * *',
   jtd_schema=load_jtd_schema(Path("/home/my-project/example-source-schema.json")),
)

Validate a set of fields

You can use field selectors to apply the same validation to a set of fields. Certain validators accept a FieldSelector object instead of a field name when specifying which field to monitor. The field selector is matched against the schema of the source, and declares an identical validator for every matching field.

When you use field selectors, the name field becomes an interpolated string. Each unique validator gets a corresponding name based on the assigned field.
You can also provide field selectors when you declare filters within a validator. Example 2 demonstrates using field selectors to declare filters in a validator.

You can only attach one field selector to a validator, either in the source field or in the filter.

For validators that use reference sources, you can use the FieldSelector.reference() object whenever the reference field is the same as the compared field. Example 3 demonstrates using reference source fields.

Example 1

Specify a declaration that creates validators for all numeric fields:

Python

from validio_sdk.resource import FieldSelector
validators.NumericValidator(
    name="mean_of_%(field)s",
    window=w,
    segmentation=sg,
    metric=NumericMetric.MEAN,
    source_field=FieldSelector(data_type=FieldDataType.NUMERIC),
)

Bash

validio code plan --diff=none
NumericValidator 'mean_of_Age' will be created
NumericValidator 'mean_of_Credit_scoring' will be created
NumericValidator 'mean_of_Number_family_members' will be created
NumericValidator 'mean_of_Working_hours_weekly' will be created
NumericValidator 'mean_of_Yearly_wage_USD' will be created
NumericValidator 'mean_of_Years_education' will be created

Plan: 6 to create, 0 to update, 0 to delete.

Example 2

Provide field selectors to declare filters within a validator. The following creates a null count validator for all nullable fields of the type String:

Python

from validio_sdk.resource import FieldSelector
validators.VolumeValidator(
    name="null_count_of_%(field)s",
    window=w,
    segmentation=sg,
    metric=VolumeMetric.COUNT,
    filter=NullFilter(field=FieldSelector(data_type=FieldDataType.STRING, nullable=True)),
)

Bash

validio code plan --diff=none
VolumeValidator 'null_count_of_favorite_date' will be created
VolumeValidator 'null_count_of_Occupation' will be created

Plan: 2 to create, 0 to update, 0 to delete.

Example 3

Create a numeric distribution validator that compares the value of the assigned field with the value of the same field in the previous window. The validators use the same source_field and reference_source_field:

Python

from validio_sdk.resource import FieldSelector
validators.NumericDistributionValidator(
    name="mean_ratio_of_%(field)s",
    window=w,
    segmentation=sg,
    metric=NumericDistributionMetric.MEAN_RATIO,
    source_field=FieldSelector(data_type=FieldDataType.NUMERIC),
    reference_source_field=FieldSelector.reference(),
    reference=validators.Reference(source=s1, window=w1, history=1, offset=1),
)

Destroy resources

To delete all resources in the project’s namespace, pass the --destroy flag to either the plan or apply command. The following example uses plan to preview the resources to delete:

Bash

validio code plan --diff=none --destroy

GcpCredential 'example-credential' will be deleted
GcpBigQuerySource 'example-source' will be deleted
TumblingWindow 'example-window' will be deleted
Segmentation 'example-segmentation' will be deleted
NumericValidator 'mean_of_Age' will be deleted
NumericValidator 'mean_of_Credit_scoring' will be deleted

Plan: 0 to create, 0 to update, 6 to delete.

Import resources

The import command adds resources to the project. Adding resources to a project means writing out Python declarations for those resources to a file in the project directory. Add the --import-namespace flag to name another namespace to import resources from. When you use the --import-namespace flag, the resources are moved to the namespace defined in validio.json.

Resources can exist in the project’s namespace without being part of the project. Run import without the --import-namespace flag to add those resources to the project.

In the following example, the import command moves the resources from other-namespace into the project’s example-namespace, as defined in validio.json. It then writes out Python declarations for those resources to the specified file, generated.py:

Bash

validio code import --output generated.py --import-namespace other-namespace

GcpCredential example-credential will be imported
GcpBigQuerySource example-source will be imported
TumblingWindow example-window will be imported
Segmentation example-segmentation will be imported
NumericValidator mean_of_Credit_scoring will be imported
NumericValidator mean_of_Age will be imported

 Plan: 6 resources will be imported from namespace 'other-namespace' to namespace 'example-namespace'

Do you want to perform these operations?
        Only 'yes' is accepted to approve
Enter a value:

The import command in the previous example moves an existing credential to the project’s namespace and generates a Python declaration for it. The attribute ignore_changes, which defaults to True, keeps the parameter values that already exist in the credential.

ImportantWhen you run the import command, resources that already exist in the project are not added again.

Specify resources to import

You can target specific resources to import with the --credential or --channel arguments:

Importing a credential automatically imports all sources, filters, validators, and so on, that are attached to the credential.
Importing a channel automatically imports all notification rules attached to the channel.

The following example moves only the gcp-prod credential, along with its sources and validators, into the current project and namespace:

Bash

validio code import -o generated.py --import-namespace other-namespace --credential gcp-prod

Import all resources

To import all resources in a namespace and move them to the project’s namespace, omit the --credential and --channel arguments. This is useful if you have been setting up Validio in the UI and want to start version controlling everything with IaC. To do this, move from the default namespace:

Bash

validio code import -o generated.py --import-namespace default

Overview

Getting Started

Validio API

Validio CLI

Validio SDK

Validio Code (IaC)

Update resources

Immutable fields

Manage source schema

Upstream schema changes

Manual schema override

Validate a set of fields

Example 1

Example 2

Example 3

Destroy resources

Import resources

Specify resources to import

Import all resources

​Update resources

​Immutable fields

​Manage source schema

​Upstream schema changes

​Manual schema override

​Validate a set of fields

​Example 1

​Example 2

​Example 3

​Destroy resources

​Import resources

​Specify resources to import

​Import all resources

Update resources

Immutable fields

Manage source schema

Upstream schema changes

Manual schema override

Validate a set of fields

Example 1

Example 2

Example 3

Destroy resources

Import resources

Specify resources to import

Import all resources