You can only configure a reference source for the following Validator types:
You can calculate a metric in reference to a ground truth source that you specify. Using a reference source you can, for example, compare data between two completely different sources. Another example is to use a reference source to create a sliding window.
The following parameters are included in the reference source config:
Select a Source as your reference data source. You can use the same Source as in the Source config step, or another Source.
The reference source uses the same Segmentation as specified in the Source config step. This assumes that the fields defined in the Source Segmentation are also available in the reference source schema.
If you use a different Source as reference source, verify that the schema has the same fields with matching data types.
Select the relevant field to act as a reference for your metric calculation.
field data typein the reference source config must match the
field data typeused in the source config.
Select a configured Window from the reference source to use in the metric calculation.
For more information, refer to Windows.
Choose by how many Windows the reference source is shifted to the past.
An offset value of
0 means that you are comparing the current data window of your source, against the current data window of your reference source.
You can choose a larger value to shift the reference source window to the past.
For example, an offset of
7 for a daily window, compares the current window of your source against the window of your reference source from
1 week ago. This assumes both the target and reference source has a daily window.
Empty reference Window?
The reference window remains empty until it can compare to datapoints from your source.
For example, if you set the
window offsetfarther back in time compared to the
lookback timespecified for a Data Warehouse source type, the reference window might remain empty until it reaches the
Choose over how many Windows the metric is calculated for the reference source.
By default the number of windows is set to
1. Larger values effectively smoothen the metric calculated for the reference. Choose a number larger than
1 to compare your current data against an aggregate of the values calculated over the total number of windows.
Optionally, select a filter to set a rule to determine which raw data is included in the metric calculation.
You can also select a filter to set a rule to determine which raw data is validated for your reference source.
For more information, refer to filters.
These examples illustrate how the metric is calculated for reference sources with different daily window configurations.
Updated 3 months ago