Volume
Monitor number of records, non-NULL values, duplicates, unique values etc.
Configuration
| Step | Required | Parameters / Options | |
|---|---|---|---|
| Validator type | ✅ | Volume | - |
| Config | ✅ | Metric | Count Percentage Duplicate count Duplicate percentage Unique count Unique percentage |
| Config | Backfill | Initialize with backfill (checkbox) | |
| Source config | ✅ | Field | No field (use record) (default) Or List of source fields |
| Source config | ✅ | Segmentation | 1. Select a configured Segmentation Or 2. Unsegmented (default) |
| Source config | ✅ | Window | Select a configured Window |
| Source config | Filter | No filter (default) Boolean Enum Null (*1) String Threshold filter | |
| Threshold | ✅ | Threshold type | Fixed threshold Dynamic threshold |
| Threshold | ✅(*2) | Operator | Less than Less than or equal Equal Not equal Greater than Greater than or equal |
| Threshold | ✅(*2) | Value | Numeric value to validate threshold on |
| Threshold | ✅(*3) | Sensitivity | Enter a numeric value |
| Threshold | ✅(*3) | Decision bounds type | Upper Lower Upper and lower (default) |
*1 Only applicable for
nullable columns.*2 Only applicable for
Fixed thresholds.*3 Only applicable for
Dynamic thresholds.
Configuration details
Field selection
For metrics Count and Percentage you can pick No field (use record) to count entire records. You can also pick a specific field to count non-NULL values in that field.
Volume Validator Configuration Wizard - Single field in Source config.
For metrics Duplicates count, Duplicates percentage, Unique count and Unique percentage you have to specify one or several fields. Specifying several fields means field values are concatenated before being counted, which is, for example, useful when validating uniqueness of composite keys.
Note: For duplicate- and unique metrics, Validio counts NULL values as any other value, unless you apply a NULL filter.
Volume Validator Configuration Wizard - Multiple fields in Source config.
Sensitivity
Higher sensitivity means that the accepted range of values is narrower, which identifies more anomalies. Conversely, lower sensitivity values imply a wider range of accepted values, which identifies fewer anomalies.
Updated about 1 year ago