Data Warehouse Sources
Connect Validio to your data warehouse to monitor tables and views for data quality issues. Validio reads your data incrementally, consolidates queries, and applies warehouse-specific optimizations to minimize cost and performance impact.
Supported Warehouses
Window Types
When defining Windows on data warehouse sources, you can define either a tumbling window or a global window. Generally, when the source includes a timestamp field, we recommend using a Tumbling window. For more information, see About Windows.
Cost and Performance
Validio minimizes warehouse load through several optimizations:
- Incremental processing: Only reads new or changed data since the last poll
- Query pushdown: Performs aggregations and filtering in your warehouse rather than transferring raw data
- Query consolidation: Combines related validations into fewer queries
- Engine-specific tuning: Optimizes query patterns for each warehouse type
Most customers see less than 1% impact on warehouse cost and performance.
Recommendations
To further reduce resource usage:
- Optimize your cursor field: Apply indexing, partitioning, or clustering to the field used for incremental reads
- Prefer tables over views: Querying views or external tables can consume significantly more resources than regular tables
- Consider field size: Large text fields are more resource-intensive to validate than smaller data types
Updated 2 months ago