About Windows
A window defines the batch size of the dataset to use for calculating and validating metrics on your source. For example, to validate an aggregate calculation (such as mean, max, or min) you must define a window to specify how much and which datapoints to include. You can then select the configured window when you create new validators or segmentations on the source.
Window Types
All window types are not valid for every source:
Window Type | Supported Source Type |
---|---|
File window | Object storage |
Fixed Batch window | Data warehouse, Data stream, Query engine |
Global window | Data warehouse, Query engine |
Tumbling window | Data warehouse, Data stream, Query Engine, Object Storage |
File Window
File windows consist of logical batches by file/BLOB. For example, one CSV file is a logical batch. For more information, see Configuring File Windows.
Fixed Batch Window
Fixed batch windows are defined by a specified number of datapoints, from a certain data-time field. For more information, see Configuring Fixed Batch Windows.
Global Window
Global windows perform a full load during every polling cycle and computes all validator metrics for the entire Source, rather than for a sequence of windows (such as for tumbling windows). Validators using a global window are evaluated based on clock-time, rather than data-time. For more information, see Configuring Global Windows.
Tumbling Window
Tumbling windows are a series of fixed-sized, non-overlapping, and contiguous time intervals. For example, hourly
or weekly
. For more information, see Configuring Tumbling Windows.
Segment Retention Policy
Segment retention period (days) is an optional setting on validator windows that sets a threshold to remove segments that may have become stale. The segment is considered stale when the last time data was processed on the segment exceeds the retention period.
The segment retention threshold is relative to the most recent segment that was processed. When not set, Validio does not clean or remove stale segments.
Updated 7 days ago