Freshness
Monitors the time elapsed since the data was last updated.
Limitations
The Freshness Validator type can only be configured on a Tumbling window.
Configuration
| Step | Required | Parameters | Options |
|---|---|---|---|
| Validator type | ✅ | Freshness | - |
| Source config | ✅ | Segmentation | 1. Select a configured Segmentation Or 2. Unsegmented (default) |
| Source config | ✅ | Window | Select a configured Window |
| Source config | Filter | No filter (default) Boolean Enum Null (*1) String Threshold filter | |
| Config | Backfill | Initialize with backfill (checkbox) |
*1 Only applicable for
nullable columns.
Configuration details
When a calculation triggers depends on the polling interval configured for the source, by using presets or a cron expression. During polling, the freshness Validator checks if any new datapoints have been added to the Source.
The resulting graph displays the time difference between latest datatime and end of window, for every window.
If your tumbling window is shorter than the polling interval, then the Validator does not calculate the Freshness metric for every window. In this case, the Freshness graph might show irregular gaps.
Updated about 1 year ago