Validio Code Workflow

This guide walks through each of the four steps in the core Validio Code workflow:

Initialize: Create a Validio Code project.
Write: Write or modify your infrastructure as code to describe the desired resources in your environment.
Plan: Preview the changes to your environment before applying them.
Apply: When you are satisfied with the program, deploy the changes to your environment.

For descriptions and examples of other scenarios, see Validio Code scenarios.

Initialize

To create a Validio Code project to manage your resources, start by scaffolding a project. A project is the directory housing the Python program responsible for managing your resources. Use the init command to create a new project. There are no special considerations for the initialized directory. It contains the files main.py and requirements.txt. In the following example, you create a project in a directory called my-project with a namespace called my-namespace:

Bash

validio code init --directory my-project --namespace my-namespace

Because it is a Python project, you can introduce any Python tools, models, dependencies, IDE, and so on.

Namespace

Always assign a unique namespace to every project that you create. To assign a namespace, add the -n or --namespace flag to the init command. The project’s namespace is recorded in the validio.json file in the project directory. For more information, see Namespaces.

Write

When writing your infrastructure as code, use the definitions provided by the Validio SDK to describe the resource types in your environment. For more information, see Validio SDK. You can use any Python-based tools at your disposal when you write a program. For example, you can reuse models, import external dependencies, and load credentials from your preferred secrets manager. To deploy a resource in your environment, create an instance of the class definition for the resource type you want to declare. For easier maintenance, you can divide your resource declarations and helper functionality across different modules, as you would in any program. To delete individual resource definitions from your program, either remove or comment out the related code.

Environment variables passed to the plan and apply commands are visible to the Validio Code program.

This example shows the contents of a working main.py file. The file consists of a full program that declares which resources to deploy into an environment:

Python

from validio_sdk.resource.credentials import GcpCredential
from validio_sdk.resource.sources import GcpBigQuerySource
from validio_sdk.resource.segmentations import Segmentation, SegmentUsage
from validio_sdk.resource.windows import TumblingWindow, WindowTimeUnit
from validio_sdk.resource.validators import NumericMetric, NumericValidator

# Declare a GCP Credential
gcp_credential = GcpCredential(name='example-credential', credential='<svc-acct>')

# Declare a BigQuery Source
big_query_source = GcpBigQuerySource(
    name='example-source',
    credential=gcp_credential,
    project='example-project',
    dataset='example-dataset',
    table='orders',
    cursor_field='created_at',
    lookback_days=30,
    schedule='0 */12 * * *',
)

# Attach a 1 hour Tumbling window to the source
window = TumblingWindow(
    name='example-window',
    source=big_query_source,
    data_time_field='event_time',
    window_size=1,
    time_unit=WindowTimeUnit.HOUR,
)

# Attach a segmentation to the source
by_gender = Segmentation(
    name='example-segmentation',
    source=big_query_source,
    fields=['Gender'],
    segment_usage=SegmentUsage.LIGHT,
)

# Set up a couple validators on the source
for field in ['Age', 'Credit_scoring']:
    NumericValidator(
        name=f'mean_of_{field}',
        window=window,
        segmentation=by_gender,
        source_field=field,
        metric=NumericMetric.MEAN,
    )

Plan

Use the plan command to preview the changes you have defined for your environment before you apply them. The preview shows any resources that are created, updated, or deleted. You can iterate between the Write and Plan steps until you are satisfied with the changes. Invoke plan whenever you want feedback on your work:

Bash

validio code plan

If you are not running plan from your project’s root directory, use the --directory flag to specify your working directory.
Use the --diff flag to choose how much of the diff to show: full, changes (the default, which shows only the changed lines), or none (a summary only).

In the following example, the output from plan shows that the program adds six resources to the environment:

Bash

validio code plan --diff=none

GcpCredential 'example-credential' will be created
GcpBigQuerySource 'example-source' will be created
TumblingWindow 'example-window' will be created
Segmentation 'example-segmentation' will be created
NumericValidator 'mean_of_Age' will be created
NumericValidator 'mean_of_Credit_scoring' will be created

Plan: 6 to create, 0 to update, 0 to delete.

Targeting individual resources

Sometimes you only want to see the plan result or apply changes for selected resources, similar to how Terraform’s target flag works. To do this, use the flags named after the resource types, such as --credential, --source, and --validator. Each flag can be specified multiple times to target multiple resources.

Bash

validio code plan \
  --credential example-credential \
  --validator mean_of_Age \
  --validator mean_of_Credit_scoring

This targets only the three specified resources and ignores any changes to the other resources. The flags work with both plan and apply.

Apply

Use the apply command to accept and perform the planned changes to your environment. Running apply also performs the Plan step, which gives you a final look at the planned changes.

Bash

validio code apply

ImportantApply has an explicit confirmation step because it is a potentially destructive operation.To skip the confirmation prompt and apply the planned changes immediately, pass the --auto-approve flag. This is useful in CI/CD workflows where you want to apply changes that are automatically merged to a branch.

The following example shows the output from plan, followed by a request for confirmation, and finally a confirmation message when apply completes:

Bash

validio code apply --diff=none

GcpCredential 'example-credential' will be created
GcpBigQuerySource 'example-source' will be created
TumblingWindow 'example-window' will be created
Segmentation 'example-segmentation' will be created
NumericValidator 'mean_of_Age' will be created
NumericValidator 'mean_of_Credit_scoring' will be created

Plan: 6 to create, 0 to update, 0 to delete.

Do you want to perform these operations?
        Only 'yes' will be accepted to approve
Enter a value: yes

...
Apply complete! Resources: 6 created, 0 updated, 0 deleted

Overview

Getting Started

Validio API

Validio CLI

Validio SDK

Validio Code (IaC)

Initialize

Directory

Namespace

Write

Plan

Targeting individual resources

Apply

​Initialize

​Directory

​Namespace

​Write

​Plan

​Targeting individual resources

​Apply

Initialize

Directory

Namespace

Write

Plan

Targeting individual resources

Apply