Reference Source Config
Compare data with a reference.
Note
You can only configure a reference source for the following Validator types:
You can calculate a metric in reference to a ground truth source that you specify. Using a reference source you can, for example, compare data between two completely different sources. Another example is to use a reference source to create a sliding window.
The following parameters are included in the reference source configuration:
Field
Select the relevant field to act as a reference for your metric calculation.
Note
The
field data type
in the reference source config must match thefield data type
used in the source config.
Window
Select a configured Window from the reference source to use in the metric calculation.
For more information, refer to Windows.
Window offset
Choose by how many Windows the reference source is shifted to the past.
An offset value of 0
means that you are comparing the current data window of your source, against the current data window of your reference source.
You can choose a larger value to shift the reference source window to the past.
For example, an offset of 7
for a daily window, compares the current window of your source against the window of your reference source from 1
week ago. This assumes both the target and reference source has a daily window.
Note
The reference window remains empty until it can compare to datapoints from your source.
For example, if you set the
window offset
farther back in time compared to thelookback time
specified for a Data Warehouse source type, the reference window might remain empty until it reaches thelookback time
.
Number of Windows
Choose over how many Windows the metric is calculated for the reference source.
By default the number of windows is set to 1
. Larger values effectively smoothen the metric calculated for the reference. Choose a number larger than 1
to compare your current data against an aggregate of the values calculated over the total number of windows.
Filter
Optionally, select a filter to set a rule to determine which raw data is included in the metric calculation.
You can also select a filter to set a rule to determine which raw data is validated for your reference source.
For more information, refer to filters.
Examples
These examples illustrate how the metric is calculated for reference sources with different daily window configurations.
Example 1
Example 2
Example 3
Updated 3 months ago