Validio supports the major data warehouses modern data teams work with today. In order for Validio to recognize new data to validate, an incremental feature/column is needed in the table to be validated. This feature has the following requirements:
- An incremental column that represents an order of records (typically timestamp)
- Values of the monotonic column of added data is greater (or later if date) than data already stored in a table
- The incremental column should not have NULL values
- If other feature values in existing rows is updated, the incremental column value needs to be updated to a greater (or later) value as well in order for Validio to validate the new values
To enable efficient ingestion into the Validio platform, make sure that the incrementing column has an index (or a similar mechanism appropriate to the data warehouse that avoids full table scans).
Any nested data to be validated needs to be flattened before ingestion into Validio. This pertains only to the feature to be validated; features that won't be validated do not need to be flattened.
Updated about 2 months ago