> ## Documentation Index
> Fetch the complete documentation index at: https://docs.validio.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Schema Detection

> Learn how Validio automatically detects and infers schemas for structured and semi-structured data sources.

Validio automatically derives a schema for every source, so your data validation rules are built on accurate schema information without manual setup. It uses three approaches:

* **Schema from metadata:** For most structured data types, Validio reads the schema from the data source metadata. For example, reading from `INFORMATION_SCHEMA` in a data warehouse.
* **Schema from inference:** When a predefined schema doesn't exist, such as for JSON or other semi-structured data types, Validio infers the schema from the existing data patterns.
* **Manual configuration:** Depending on the data source and types, you can manually configure the schema by selecting fields, including nested fields, for validation.

![The source schema list showing detected fields and their profiling results](https://mintcdn.com/validio/36lh-6_uXUsulPvT/images/docs/schema-and-profiling/760-schema-profiling-tab.png?fit=max\&auto=format\&n=36lh-6_uXUsulPvT\&q=85\&s=42123276883b9c46bb6e8d3d8db12005)

Source schema list with profiling results

## Supported complex data types

In addition to structured data, Validio supports semi-structured and other complex data types. You can select these fields or nested fields when you [configure a source](https://docs.validio.io/v11.1/docs/sources.md#configuring-a-source).

### Data warehouses

| Source system | Supported complex data types |
| :- | :- |
| Athena | ARRAY, MAP, STRUCT |
| ClickHouse | Tuple, Named Tuple, Nested, Array |
| Databricks | STRUCT, ARRAY, VARIANT |
| Google BigQuery | JSON, ARRAY, STRUCT |
| PostgreSQL | JSON, JSONB, Array |
| Redshift | SUPER |
| Snowflake | ARRAY, OBJECT, VARIANT |

### Data streams

| Source system | Supported complex data types |
| :- | :- |
| Kafka | JSON, Protobuf, Avro |
| Kinesis | JSON, Protobuf, Avro |
| Pub/Sub | JSON, Protobuf, Avro |

> [!NOTE]
>
> **JSONPath expressions and arrays**
>
> Currently Validio does not support data validation within an array. However, you can validate the size of an array.
>
> Validio uses JSONPath expressions to represent data structures. For each array, Validio adds a computed numeric field named `some_array.length()` to represent the size of an array. For more information, see [Semi-structured Data Syntax](https://docs.validio.io/v11.1/developer/getting-started/semi-structured-data-syntax.md).

## Understanding nullable fields

If the automatically inferred schema doesn't match your expectations for incoming data, you can modify the nullability settings and data types to better reflect your data structure.

* Check the `Nullable` option to include datapoints with `NULL` values in validation.
* When unchecked, datapoints with `NULL` values in that field will be excluded from validator metrics.

This gives you control over how missing or null data affects your validation results.

## When automatic detection fails

![A schema detection error offering the option to upload a sample data file](https://mintcdn.com/validio/36lh-6_uXUsulPvT/images/docs/schema-and-profiling/schema-detection-upload-sample-data.png?fit=max\&auto=format\&n=36lh-6_uXUsulPvT\&q=85\&s=00b871139b29a0c49c316093ef006c99)

Schema detection error with option to Upload sample data file.

Schema inference may encounter issues in these scenarios:

* **Large tables:** Timeout occurs when source tables are too large to process efficiently
* **Unknown data types:** Cannot determine appropriate data types for schema fields
* **Mixed data types:** Semi-structured data with inconsistent data types across rows

### Upload a JSON sample

When automatic detection fails, upload a JSON sample file to help Validio understand your schema structure.

Sample data format:

```json title="JSON" theme={null}
{
  "date": "2025-01-01",
  "user_id": 123,
  "user_profile": {
    "age": 20,
    "name": "Bob"
  }
}
```

> [!WARNING]
>
> **Case Sensitivity:** The properties (fields and values) used in the uploaded sample data file must match the case conventions of your data source. For example, Snowflake defaults to uppercase, so your sample data should use uppercase field names.

## Schema change validation

Validio automatically validates schema changes for structured data in data warehouses and object storage files. Validio runs schema checks hourly, and reports detected changes as source errors, not incidents. You can review them on the source's **Schema** tab, and receive notifications for them with a notification rule of the **Schema change** type. For more information, see [About Source Errors](https://docs.validio.io/v11.1/docs/incidents-and-errors/errors.md#types-of-errors) and [Adding Notification Rules](https://docs.validio.io/v11.1/docs/notifications/notification-rules.md).