Numeric Distribution Validator

Validator overview

The Numeric Distribution validator compares statistical properties of numeric fields between two datasets to ensure stability over time. You can configure numeric distribution validators to monitor the following metrics:

Metric Options	Description
Maximum Ratio	Validate the ratio of the maximum value between two datasets.
Mean Ratio	Validate the ratio of the mean value between two datasets.
Minimum Ratio	Validate the ratio of the minimum value between two datasets.
Relative Entropy	Validates the difference in distribution between two datasets
Standard Deviation Ratio	Validate the ratio of the standard deviation value between two datasets.
Sum Ratio	Validate the ratio of the sum value between two datasets.

Numeric distribution validators support reference source configuration for data validation. For more information, see Reference Source Validation. For an overview of all validator types, see Validator Types.

Relative entropy

You can use relative entropy to validate distribution shifts in your data over time, or to compare the distributions of two data sets. Relative entropy is presented as a percentage where:

0% means identical empirical distributions.

100% means maximal difference in empirical distributions.

In Validio, relative entropy is adapted from the implementation of the Kullback - Leibler divergence.

Ratio metrics

Calculates the ratio of mean, sum, maximum, minimum, or standard deviation between the source and reference datasets:

\text{Ratio} = \frac{\text{Source metric}}{\text{Reference metric}}

Metric configuration parameters

Configure the validator metric calculation with the the following parameters:

Parameters	Description
Metric	Select the metric to calculate. See the Metric Options table.
Field	Select a source field from a list of available fields with numeric data type.
Reference field	Select a reference field from a list of available fields with numeric data type.
Filter	(Optional) Select from a list of filters or create a new filter to specify which records to include.
Reference filter	(Optional) Select from a list of filters or create a new filter to specify which reference records to include.
Window	Select from a list of windows or create a new window to specify how to aggregate the data.
Reference window offset	Enter a number to specify the how many windows to shift back in time to compare against the current window.
Number of reference windows	Enter a number to specify the number of windows to include in the aggregation.
Segmentation	(Optional) Apply segmentation to analyze data in separate groups. Default is `Unsegmented`.

Introduction

Using Validio

Validio Tutorials

AI Features

Credentials

Sources

Validators

Catalog

Lineage

Incidents & Errors

Notifications

Integrations

Schema & Profiling

Filters

Windows

Segmentation

Thresholds

Access & Settings

Deployment

Security

Other

Numeric Distribution

Validator overview

Relative entropy

Ratio metrics

Metric configuration parameters

​Validator overview

​Relative entropy

​Ratio metrics

​Metric configuration parameters

Validator overview

Relative entropy

Ratio metrics

Metric configuration parameters