Configuring File Windows
File windows consist of logical batches by file/BLOB. For example, one CSV file is a logical batch. File windows are only supported for object storage sources.
To add a File window to a source,
- Navigate to the source's details page and click the Windows tab.
- Click + New window.
- Under Select Type, choose File window.
- Under Configuration,
- Select a Data-time field to use for creating the windows. For more information see Configuration Parameters.
- Enter a Name for the window, or click Generate name to automatically create one based on your configuration.
- (Optional) Expand Advanced to configure the segment retention period.
- Click Create Window.
After creating a window, you can add validators and segmentations to use the new window. For more information, see Configuring a Validator and Configuring Segmentation.
Configuration Parameters
Displayed Time Field
When configuring windows, a time field is used. The format of the time field is converted to UTC in Validio, but the graphs will always display times in your system's timezone.
| Field | Value | Description |
|---|---|---|
| Name | String | Identifier for the window, used when configuring validators or segmentation. |
| Data-time field | Field name | Identifier for the field used to create the Window. |
| Segment retention period (days) | Number | (Optional) Maximum number of days to keeps segments when new data has not been found. |
Monitoring Data Drift
File window datasets are often used for distribution shift validation, such as in ML use cases to monitor data drifts. If you use a production training dataset as reference dataset, as new data is collected, you can monitor distribution shift metrics between the reference dataset and the newly collected dataset.
For information on numeric reference metrics, such as relative entropy, refer to the Numeric distribution or Categorical distribution Validator types.
Updated 7 days ago