Volume
Monitor number of records, non-NULL values, duplicates, unique values etc.
Configuration
Step | Required | Parameters / Options | |
---|---|---|---|
Validator type | ✅ | Volume | - |
Config | ✅ | Metric | Count Percentage Duplicate count Duplicate percentage Unique count Unique percentage |
Config | Backfill | Initialize with backfill (checkbox) | |
Source config | ✅ | Field | No field (use record) (default) Or List of source fields |
Source config | ✅ | Segmentation | 1. Select a configured Segmentation Or 2. Unsegmented (default) |
Source config | ✅ | Window | Select a configured Window |
Source config | Filter | No filter (default) Boolean Enum Null (*1) String Threshold filter | |
Threshold | ✅ | Threshold type | Fixed threshold Dynamic threshold |
Threshold | ✅(*2) | Operator | Less than Less than or equal Equal Not equal Greater than Greater than or equal |
Threshold | ✅(*2) | Value | Numeric value to validate threshold on |
Threshold | ✅(*3) | Sensitivity | Enter a numeric value |
Threshold | ✅(*3) | Decision bounds type | Upper Lower Upper and lower (default) |
*1 Only applicable for
nullable columns
.*2 Only applicable for
Fixed thresholds
.*3 Only applicable for
Dynamic thresholds
.
Configuration details
Field selection
For metrics Count
and Percentage
you can pick No field (use record)
to count entire records. You can also pick a specific field to count non-NULL
values in that field.
For metrics Duplicates count
, Duplicates percentage
, Unique count
and Unique percentage
you have to specify one or several fields. Specifying several fields means field values are concatenated before being counted, which is, for example, useful when validating uniqueness of composite keys.
Note: For duplicate- and unique metrics, Validio counts NULL
values as any other value, unless you apply a NULL filter
.
Sensitivity
Higher sensitivity means that the accepted range of values is narrower, which identifies more anomalies. Conversely, lower sensitivity values imply a wider range of accepted values, which identifies fewer anomalies.
Updated about 1 year ago