Configuring Tumbling Windows
Tumbling windows are a series of fixed-sized, non-overlapping, and contiguous time intervals. For example, hourly or weekly. Tumbling windows are supported on all sources.
To add a Tumbling window to a source,
- Navigate to the source's details page and click the Windows tab.
- Click + New window.
- Under Select Type, choose the Tumbling window.
- Under Configuration,
- Complete the required fields: Data-time field, Window size, and Historic data (maximum). For more information, refer to the Configuration Parameters table.
- Enter a Name for the window, or click Generate name to automatically create one based on your configuration.
- (Optional) Under Advanced, you can configure partition filtering, segment retention, and disable window timeout.
- Click Create Window.
After creating a window, you can add validators and segmentations to use the new window. For more information, see Configuring a Validator and Configuring Segmentation.
Configuration Parameters
Displayed Time Field
When configuring windows, a time field is used. The format of the time field is converted to UTC in Validio, but the graphs will always display times in your system's timezone.
Field | Values | Description |
|---|---|---|
Name | String | Identifier for the window, used when configuring validators or segmentation. |
Data-time field | Field name | The name of the field that references the timestamp associated with each record or row in the data source. See Tumbling Window and Time Formats. |
Window size | Number and unit | Length of the window in the selected time unit (such as minute, hour, day, week, month). For example, |
Historic data (maximum) | Number and unit | The maximum lookback period, which defines how far back in time Validio starts reading from the source when backfilling the validator. A higher segment usage can limit the amount of backfill. For more information, Segment Usage Options. |
Partition field | Field name | (Optional) For partitioned tables, add a filter on the partition column. For more information, see Partition Filter. |
Partition lookback | Number and unit | (Optional) Defines are how far back in time relative to the window start time to look at data. |
Segment retention period (days) | Number | (Optional) Maximum number of days to keeps segments when new data has not been seen. |
Disable window timeout | True | (Optional) Set to true if the window should be automatically closed without considering the most recent data-time. |
Minimum Window Size
The smallest allowed size for a tumbling window is:
- Data Warehouses, Query Engines, Object Storages: 30 minutes
- Data Streams: 10 minutes
Tumbling Windows and Time Formats
Validators with a Tumbling window always follow data time, which means that the time used in graphs and metrics represents the time of the dataset (such as, the time of the data in the warehouse). For tumbling windows, the time is also truncated depending on the window size. For example,
- Daily windows will always be from 00:00 UTC to 24:00 UTC.
- 12 hour windows will be 00:00 UTC to 12:00 UTC and 12:00 to 24:00 UTC.
- Hourly windows will always start on the hour: 00:00 UTC, 01:00 UTC, 02:00 UTC, and so on.
- 30 minute windows will truncate to every whole and half hour: 02:00 UTC, 02:30 UTC, 03:00 UTC, and so on.
Validio polls for new data on a schedule defined by the polling interval configured on the source. A tumbling window closes (finishes the computation and updates the graph) as soon as all the data for that window has arrived.
Validio determines that all the data has arrived in two ways:
- New data arrives with a timestamp that is after the window end time. For example, if the window is from 13:00 to 14:00 and there is new data at 14:30, Validio assumes all data between 13:00 - 14:00 has arrived.
- When
Disable window timeoutis selected and data exists within a window. For example, if the window is from 13:00 to 14:00, and a poll for new data found a data point with a time 13:30, that window will close.
In general, the graphs for validators will display data points until the last datapoint, and then stick at whenever the last window closes because the graphs do not have new data points to show unless new data arrives.
Freshness Validator Graph
The exception is seen in Freshness validators, where the graph will continue to fill in intermediate data points as time passes even when no data arrives. For more information, see Freshness Validator.
Partition Filter
For a partitioned table, this provides configuration to add a qualifying filter on the value of the partitioning column. When the the configured data-time field of the tumbling window is not the same as the partitioning column of the table, a partition filter helps prune out irrelevant partitions in queries.
Updated 5 days ago