Validio reads data incrementally, which requires that you specify a
cursor field (incremental field) and
lookback time. The
lookback time indicates how far back in time Validio starts reading data from your source.
cursor field is a timestamp which represents when data was updated or added:
cursor fieldmust be a timestamp.
cursor fieldshould not include
NULLvalues. Any records where the cursor field is
Validio recommends that you provide a
cursor field that represents when data was updated, rather than added. This is to make sure that all records are part of the validation, even in cases of late data arrival.
When defining Windows on your Data Warehouse sources, Validio recommends using either a Tumbling window or a Global window. Fixed batch windows are supported, but may not perform as expected. For more information, see About Windows.
Validio applies advanced optimizations when it reads and processes data for validations. For example, Validio processes data incrementally, leverages pushdown, consolidates related queries, and optimizes queries for each query engine.
Typically, our customers notice a very low (~1 %) impact on performance and/or cost on their Data Warehouse, when they use Validio.
When you validate data in a Data Warehouse, we recommend that you:
- Apply optimizations, such as indexing, partitioning, and clustering, on the specified cursor field.
- Be aware that querying views or external tables can consume significantly more resources than querying regular tables.
- Consider the size of the fields you validate. Validating fields with much data, such as text blogs, is more resource intensive than validating fields with less data.
Updated 21 days ago