HomeDocumentationChangelog
HomeDemoContact
HomeDemoContact

Numeric anomaly

Identify numeric anomalies in your data with Machine Learning algorithms.

📘

Destination

Optionally, create a Destination to write out datapoints to a Destination, before you configure the numeric anomaly Validator.

Validator overview

Validate individual field values for every datapoint Validio reads, by comparing the field value in a reference source. Dynamic anomaly bounds are configured with the sensitivity parameter.

The Numeric anomaly validator identifies anomalies based on either count or percentage:

  • Count: Counting how many datapoints are identified as an anomaly in each window.
  • Percentage: Counting the share of datapoints that are identified as an anomaly in each window.

Configuration

StepRequiredParametersOptions
Validator type✅Numeric anomaly-
Config✅MetricCount
Percentage
Config✅SensitivityEnter a numeric value
ConfigDestination1. No Destination (default)

Or

2. Select a pre-configured Destination
ConfigAdvanced configMinimum absolute difference
Minimum number of reference datapoints
Minimum relative difference percent
ConfigBackfillInitialize with backfill (checkbox)
Source fields✅FieldList of source fields with numeric data types
Source config✅Segmentation1. Select a configured Segmentation

Or

2. Unsegmented (default)
Source config✅WindowSelect a configured Window
Source configFilterNo filter (default)
Boolean
Enum
Null (*1)
String
Threshold filter
Reference source config✅SourcesSelect a Source to use as reference source
Reference source config✅FieldSelect a valid field from your reference source
Reference source config✅WindowSelect a configured Window
Reference source config✅Window offsetSelect how many Windows you want to offset by
Reference source config✅Number of WindowsSelect how many Windows to include
Reference source configFilterNo filter (default)
Enum
Null (*1)
String
Threshold Filter
Threshold✅Threshold typeFixed threshold
Dynamic threshold
Threshold✅(*2)OperatorLess than
Less than or equal
Equal
Not equal
Greater than
Greater than or equal
Threshold✅(*2)ValueNumeric value to validate threshold on
Threshold✅(*3)SensitivityEnter a numeric value
Threshold✅(*3)Decision bounds typeUpper
Lower
Upper and lower (default)

*1 Only applicable for nullable columns.

*2 Only applicable for Fixed thresholds.

*3 Only applicable for Dynamic thresholds.

Configuration details

Sensitivity

Higher sensitivity means that the accepted range of values is narrower, which identifies more anomalies. Conversely, lower sensitivity values imply a wider range of accepted values, which identifies fewer anomalies.

Destination

The Destination parameter allows writing out data to a specified Destination for follow-up inspection. This means that data caught by the filter in this Validator can be stored in a separate table or bucket.

Advanced config

Minimum absolute difference:

The minimum absolute difference between the field value and the mean of the reference distribution for the datapoint to be considered an anomaly.

For example, if set to 10, the difference between the mean of the reference distribution and the datapoint being validated must be greater than 10, and be outside the dynamic bounds to be considered an anomaly. Essentially, this is an ignore any incidents within the difference parameter.

Minimum number of reference datapoints:

Minimum number of datapoints in reference source before triggering a metric calculation.

Minimum relative difference percent:

Minimum difference for datapoints to be considered an anomaly expressed in relative terms, divides absolute difference with absolute of the mean of the reference data.

For example, if the mean of the reference distribution is 10, and user sets 10% as parameter value, then, datapoints falling between 9 and 11 are not considered anomalies.

We recommend that you use this option instead of minimum absolute difference, when you are more interested in the relative difference to the reference mean, than the absolute difference.

**Numeric anomaly** Validator Configuration Wizard - Config.

Numeric anomaly Validator Configuration Wizard - Config.

Reference source

For information on how you configure the reference source, refer to Reference Source.