> ## Documentation Index
> Fetch the complete documentation index at: https://docs.validio.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Validio Code Scenarios

> Explore Validio Code scenarios including updating resources, managing schemas, importing resources, and destroying definitions.

The following sections provide examples for different scenarios using Validio Code:

* **Update resources:** Update the configuration of a resource.
* **Manage source schema:** Fetch upstream schema changes or manually override the schema.
* **Validate a set of fields:** Apply the same validation to a set of fields, or declare filters within a validator.
* **Destroy resources:** Delete all resources in the project's namespace.
* **Import resources:** Generate code for resources that are not present in the project.

## Update resources

Resources are the configured components in your Validio environment, and include channels, credentials, sources, validators, windows, and so on. You can update the configuration of a resource by editing the resource's declaration in the code.

The [Write step of the Validio Code workflow](https://docs.validio.io/v10.3/developer/validio-code/workflow.md#write) includes a `main.py` example with a 1-hour tumbling window. In the following example, you edit the window definition to disable the window timeout:

```python title="Python" theme={null}
window = TumblingWindow(
    name='example-window',
    source=big_query_source,
    data_time_field='event_time',
    window_size=1,
    time_unit=WindowTimeUnit.HOUR,
    # Updated disabling window timeout from False to True
    window_timeout_disabled=True
)
```

Then, invoke the `plan` command to display the pending update:

```bash title="Bash" theme={null}
validio code plan

TumblingWindow 'example-window' will be updated
~ TumblingWindow = {
~   window_timeout_disabled = False => True
~ }


Plan: 0 to create, 1 to update, 0 to delete.
```

### Immutable fields

Some fields are immutable because changing them would drastically change the behavior of the resource and its monitoring results. These fields are not exposed in the user interface, and changing them requires you to remove the resource and set up a new resource with the new settings. For example, the window on a validator is immutable because changing the window would invalidate all historical metrics for that validator.

With IaC, changing an immutable field in the manifest automatically removes the old resource and creates the new one for you. If the resource to be recreated has child resources, such as when you change an immutable field on a source, all resources with that source as a parent (such as validators, windows, and segmentations) are also deleted and recreated.

> [!WARNING]
>
> **Important**
>
> When a source or validator is deleted, all historically processed metrics are also deleted. Once the new configuration is created, you need to run a backfill to recover the historical data.

In IaC, you can review the summary of resources being removed or marked as replaced. The following example shows the output when you change the immutable `time_unit` field on a tumbling window that is used by two validators:

```bash title="Bash" theme={null}
validio code plan

TumblingWindow 'tumbling_window' will be replaced: immutable field 'time_unit' was updated
~ TumblingWindow(
~   time_unit = WindowTimeUnit.DAY => WindowTimeUnit.HOUR
~ )

NumericValidator 'mean_of_age' will be replaced: cascading replacement from TumblingWindow(name="tumbling_window")
- NumericValidator(
-   name = mean_of_age
-   window_name = tumbling_window
-   ...
- )

NumericValidator 'mean_of_working_hours_weekly' will be replaced: cascading replacement from TumblingWindow(name="tumbling_window")
- NumericValidator(
-   name = mean_of_working_hours_weekly
-   window_name = tumbling_window
-   ...
- )


Plan: 3 to create, 0 to update, 3 to delete.
```

All three resources are marked "will be replaced", and the plan summary shows "3 to create" and "3 to delete", even though the diff shows an update for the window and only deletes for the validators.

## Manage source schema

A source's configuration includes a set of expected fields and data types, which are defined by the schema. Usually you do not interact with or modify the schema, because Validio automatically detects or infers the schema for all source types. However, you can intervene to manually update the schema to do the following:

* Capture upstream schema changes
* Override the inferred schema with a modified version

### Upstream schema changes

When the schema of the upstream data source changes, for example because of changes in the warehouse table, stream messages, or object storage file, it might add new fields that you would like to monitor.

Pass the `--update-schema` flag to the `plan` or `apply` command to fetch the latest schema.

> [!NOTE]
>
> You can pass the `--update-schema` flag multiple times to check several sources for updates, or use the `--update-all-schemas` flag to check for schema updates on all sources.

The following example checks for upstream schema updates for the source and reports any changes:

```bash title="Bash" theme={null}
validio code plan --diff=changes --update-schema example-source

No changes. Configuration is up-to-date!
```

### Manual schema override

You can manually override the inferred schema with a modified version, with the following steps:

1. Download the inferred schema.
2. Edit the downloaded schema file.
3. Pass the contents of the modified schema file as a parameter to the source.

The `plan` and `apply` commands detect a change to the source's schema whenever the contents of the file change. For more information about interpreting and modifying the schema file, see the [JSON Type Definition](https://jsontypedef.com/docs/jtd-in-5-minutes/) documentation.

The following example shows how to manually override an inferred schema with a modified schema file. First, download the inferred schema:

```bash title="Bash" theme={null}
validio sources infer-schema bigquery \
  --credential-id example-credential \
  --project example-project \
  --dataset example-dataset \
  --table orders \
  --filename example-source-schema.json
```

Then, edit the downloaded file and pass the contents of the modified schema as a parameter to the source:

```python title="Python" theme={null}
from pathlib import Path
from validio_sdk import load_jtd_schema

# Use a manually provided schema instead for the source.
big_query_source = GcpBigQuerySource(
   name="example-source",
   credential=gcp_credential,
   project='example-project',
   dataset='example-dataset',
   table='orders',
   cursor_field='created_at',
   lookback_days=30,
   schedule='0 */12 * * *',
   jtd_schema=load_jtd_schema(Path("/home/my-project/example-source-schema.json")),
)
```

## Validate a set of fields

You can use field selectors to apply the same validation to a set of fields.

Certain validators accept a `FieldSelector` object instead of a field name when specifying which field to monitor. The field selector is matched against the schema of the source, and declares an identical validator for every matching field.

* When you use field selectors, the name field becomes an interpolated string. Each unique validator gets a corresponding name based on the assigned field.
* You can also provide field selectors when you declare filters within a validator. Example 2 demonstrates using field selectors to declare filters in a validator.

> [!NOTE]
>
> You can only attach one field selector to a validator, either in the source field or in the filter.

For validators that use reference sources, you can use the `FieldSelector.reference()` object whenever the reference field is the same as the compared field. Example 3 demonstrates using reference source fields.

### Example 1

Specify a declaration that creates validators for all numeric fields:

```python title="Python" theme={null}
from validio_sdk.resource import FieldSelector
validators.NumericValidator(
    name="mean_of_%(field)s",
    window=w,
    segmentation=sg,
    metric=NumericMetric.MEAN,
    source_field=FieldSelector(data_type=FieldDataType.NUMERIC),
)
```

```bash title="Bash" theme={null}
validio code plan --diff=none
NumericValidator 'mean_of_Age' will be created
NumericValidator 'mean_of_Credit_scoring' will be created
NumericValidator 'mean_of_Number_family_members' will be created
NumericValidator 'mean_of_Working_hours_weekly' will be created
NumericValidator 'mean_of_Yearly_wage_USD' will be created
NumericValidator 'mean_of_Years_education' will be created

Plan: 6 to create, 0 to update, 0 to delete.
```

### Example 2

Provide field selectors to declare filters within a validator. The following creates a null count validator for all nullable fields of the type `String`:

```python title="Python" theme={null}
from validio_sdk.resource import FieldSelector
validators.VolumeValidator(
    name="null_count_of_%(field)s",
    window=w,
    segmentation=sg,
    metric=VolumeMetric.COUNT,
    filter=NullFilter(field=FieldSelector(data_type=FieldDataType.STRING, nullable=True)),
)
```

```bash title="Bash" theme={null}
validio code plan --diff=none
VolumeValidator 'null_count_of_favorite_date' will be created
VolumeValidator 'null_count_of_Occupation' will be created

Plan: 2 to create, 0 to update, 0 to delete.
```

### Example 3

Create a numeric distribution validator that compares the value of the assigned field with the value of the same field in the previous window. The validators use the same `source_field` and `reference_source_field`:

```python title="Python" theme={null}
from validio_sdk.resource import FieldSelector
validators.NumericDistributionValidator(
    name="mean_ratio_of_%(field)s",
    window=w,
    segmentation=sg,
    metric=NumericDistributionMetric.MEAN_RATIO,
    source_field=FieldSelector(data_type=FieldDataType.NUMERIC),
    reference_source_field=FieldSelector.reference(),
    reference=validators.Reference(source=s1, window=w1, history=1, offset=1),
)
```

## Destroy resources

To delete **all resources** in the project's namespace, pass the `--destroy` flag to either the `plan` or `apply` command.

The following example uses `plan` to preview the resources to delete:

```bash title="Bash" theme={null}
validio code plan --diff=none --destroy

GcpCredential 'example-credential' will be deleted
GcpBigQuerySource 'example-source' will be deleted
TumblingWindow 'example-window' will be deleted
Segmentation 'example-segmentation' will be deleted
NumericValidator 'mean_of_Age' will be deleted
NumericValidator 'mean_of_Credit_scoring' will be deleted

Plan: 0 to create, 0 to update, 6 to delete.
```

## Import resources

The `import` command adds resources to the project. Adding resources to a project means writing out Python declarations for those resources to a file in the project directory.

Add the `--import-namespace` flag to name another namespace to import resources from. When you use the `--import-namespace` flag, the resources are moved to the namespace defined in `validio.json`.

> [!NOTE]
>
> Resources can exist in the project's namespace without being part of the project. Run `import` without the `--import-namespace` flag to add those resources to the project.

In the following example, the `import` command moves the resources from `other-namespace` into the project's `example-namespace`, as defined in `validio.json`. It then writes out Python declarations for those resources to the specified file, `generated.py`:

```bash title="Bash" theme={null}
validio code import --output generated.py --import-namespace other-namespace

GcpCredential example-credential will be imported
GcpBigQuerySource example-source will be imported
TumblingWindow example-window will be imported
Segmentation example-segmentation will be imported
NumericValidator mean_of_Credit_scoring will be imported
NumericValidator mean_of_Age will be imported

 Plan: 6 resources will be imported from namespace 'other-namespace' to namespace 'example-namespace'

Do you want to perform these operations?
        Only 'yes' is accepted to approve
Enter a value:
```

The `import` command in the previous example moves an existing credential to the project's namespace and generates a Python declaration for it. The attribute `ignore_changes`, which defaults to `True`, keeps the parameter values that already exist in the credential.

> [!WARNING]
>
> **Important**
>
> When you run the `import` command, resources that already exist in the project are not added again.

### Specify resources to import

You can target specific resources to import with the `--credential` or `--channel` arguments:

* Importing a credential automatically imports all sources, filters, validators, and so on, that are attached to the credential.
* Importing a channel automatically imports all notification rules attached to the channel.

The following example moves only the `gcp-prod` credential, along with its sources and validators, into the current project and namespace:

```bash title="Bash" theme={null}
validio code import -o generated.py --import-namespace other-namespace --credential gcp-prod
```

### Import all resources

To import all resources in a namespace and move them to the project's namespace, omit the `--credential` and `--channel` arguments. This is useful if you have been setting up Validio in the UI and want to start version controlling everything with IaC. To do this, move from the `default` namespace:

```bash title="Bash" theme={null}
validio code import -o generated.py --import-namespace default
```