Query Engine Sources
Validio supports reading data from Query Engines.
General considerations
Validio reads data incrementally, which requires that you specify a cursor field
(incremental field) and lookback time
. The lookback time
indicates how far back in time Validio starts reading data from your source.
A cursor field
is a timestamp which represents when data was updated or added:
- The
cursor field
must be a timestamp. - The
cursor field
should not includeNULL
values. Any Records where the cursor field isNULL
are ignored.
Validio recommends that you provide a cursor field
that represents when data was updated, rather than added. This is to make sure that all records are part of the validation, even in cases of late data arrival.
Cost and performance
Validio applies advanced optimizations when it reads and processes data for validations. For example, Validio processes data incrementally, leverages pushdown, consolidates related queries, and optimizes queries for each query engine.
When you validate data from a Query Engine, we recommend that you:
- Apply optimizations, such as compressing, partitioning, and converting the underlying data into columnar formats.
- Consider the size of the fields you validate. Validating fields with much data, such as text blogs, is more resource intensive than validating fields with less data.
Updated 10 months ago