Thresholds define the conditions required for a validator to consider a metric to be a data quality incident or anomaly. When the validator detects data that breaches the defined threshold, it creates an incident that you can inspect on the validator details page. Optionally, you can define rules to notify you about identified incidents and route the notifications to different channels, such as Slack or webhooks.

When creating a Validator, you must configure a threshold to identify data quality incidents. Validio supports the following types of thresholds:

• Dynamic Threshold–Automatically calculates thresholds for numeric metrics based on statistical methods and allows you to adjust the sensitivity, which is the range of accepted threshold values. For example, you can define a dynamic threshold to track the daily average of sales to detect anomalies. For more information, see Configuring Dynamic Thresholds.
• Fixed Threshold–Performs comparison operations between the metric and a specified numeric threshold. For example, you can define a fixed threshold to check that no values in the field "Age" are less than zero. For more information, see Configuring Fixed Thresholds.

For more information about review validator incidents, see About Validator Details and About Validator Incidents. For information on how to configure rules and channels to receive notifications when validator incidents occur, see About Notifications.

Configuring Dynamic Thresholds

Dynamic Thresholds use a combination of smart algorithms to automatically detect anomalies in your data. The threshold model infers trends, seasonality, and peaks, and also adapts to shifts in your data. It learns from historical data and is trained on new data, continuously improving as more data is read.

When applied to a backfilled source, the dynamic thresholds can quickly detect upcoming anomalies without any training period. This means you get incidents and insight immediately, even if you lack the domain knowledge to create appropriate thresholds.

Dynamic thresholds will continuously track and automatically update when it detects shifts in seasonality and trends. You can use dynamic thresholds to monitor sources where you expect changes in your data over time. For more information, see Seasonality Detection.

The following table lists the parameters for configuring a dynamic threshold:

Sensitivity

Sensitivity defines the accepted range of values for the dynamic threshold.

• Higher sensitivity (lower threshold)–Means that the accepted range of values is more narrow, and the model will identify more data quality incidents or anomalies, leading to more alerts. Higher sensitivity is best suited for your most important tables.
• Lower sensitivity (higher threshold)–Implies a wider range of accepted values, resulting in fewer incidents and alerts. Lower sensitivity is ideal for less important tables that have historically produced noisy incidents.

Setting the right sensitivity is often an iterative process to find a balance between false positives and alert fatigue versus false negatives and missing real errors. The typical starting sensitivity value for testing is between 2 and 3. The default sensitivity in Validio is 2.0.

The following table maps the numeric value of Validio sensitivity presets to standard deviations:

Decision bounds type

The decision bound type on the dynamic threshold specifies whether the boundaries for anomaly detection are double or single-sided:

• Upper and lower–Detects both upper and lower anomalies.
• Upper–Treats only upward deviations as anomalies. For example, this is the default for freshness validators. You do not want to be alerted about too fresh data but rather when your data is late.
• Lower–Treats only downward deviations as anomalies.

Seasonality Detection

Dynamic thresholds can automatically adapt to seasonality patterns that appear in your data which is related to the calendar. You do not have to enable or configure this feature. When there is enough evidence in your data to support the pattern detection, the dynamic threshold will adapt and not trigger an incident if it is caused by the seasonality.

• Calendric Seasonality–Seasonal patterns can appear in your data due to the calendar. Calendric seasonality can relate to business processes and cycles where work may be planned and reviewed in regular cycles that may be weekly, bi-weekly, or monthly, and this behavior is reflected in your data. One example of calendric seasonality is recognizing that a Volume validator returns 0 on all days except the days when the pipeline runs and ingests data.

Configuring Fixed Thresholds

Fixed thresholds perform comparison operations between numeric metrics and a specified value. For example, you can define a fixed threshold to check that no values in the field “Age” are less than zero. When the validator detects data matching the conditions, it creates an incident and sends an alert to any configured channels.

The following table lists the parameters for configuring a fixed threshold: