Configure a Source to authenticate Validio and define a data source to validate. Supported source types include the Data Warehouses, Object storage, and Data streams listed in this table:
We want to make Validio the world’s best Deep Data Observability platform, and would love your feedback on what Sources to support. Let us know at [email protected].
Validio defines a schema for every source, either based on metadata or inference.
Schema from metadata:
Validio reads the schema from the metadata in the data source, for example, from
INFORMATION_SCHEMA in a Data Warehouse. This is true for most structured data types.
Schema from inference:
Validio infers the schema from the existing data when no pre-defined schema exists. This is true for most semi-structured data types, for example
JSON. Depending on your Source type, it might take a few seconds to infer the schema.
Validio can only infer schema when data exists in the source.
Validio will automatically detect schema changes for structured data types in Data warehouses and files in Object storage.
In addition to structured data, Validio supports semi-structured and other complex data types. You can select these fields or certain nested fields when you configure a Source.
Validio uses JSONPath to represent data structures.
For example, the JSONPath expression
some_array.length()represents the size of an array.
|Source system||Data type|
|BigQuery||JSON, RECORD (NULLABLE and REPEATED)|
|PostgreSQL||JSON, JSONB, array types|
|Snowflake||ARRAY, OBJECT, VARIANT|
|Pub/Sub||JSON, protobuf, Avro|
|Pub/Sub Lite||JSON, protobuf, Avro|
Currently, Validio does not support data validation within an array.
You can validate the size of an array. For each array, Validio adds a computed field named
some_array.length(), which you can validate as a numeric field.
Updated 5 months ago