Validators can be used to validate data for a huge number of use cases. Some recipes for common and useful validations are described below.

👍

Recommendation

For more customizability we recommend exploring the Custom SQL recipes and the SQL filter recipes.

---

Ensure data is up to date, for example data from every supplier

Validates freshness, for every segment defined by a segmentation field

Note that polling interval and window size have a significant impact on freshness validation.

For instance, consider supplier updates occurring around XX:40 every hour, and you have set the window size to hourly. Freshness is assessed against the end of the window, making the expected freshness 20 minutes. If one supplier's data is updated late, say 09:55, the freshness will be reduced to 5 minutes, which is captured by the dynamic thresholds as an anomaly.

Additionally, in this scenario, setting the polling interval to occur every whole hour offers two advantages. First, it enables immediate freshness evaluation right after polling, facilitating prompt notifications. Second, it helps avoid late arrivals. For instance, data with a timestamp of 10:40 arriving at 11:02 will not affect freshness as the poll closes the window at 11:00.

🚧

Caution

A very frequent polling interval can impact cost and performance, depending on, for example, amount of data and other validators running on the same Source.

---

Validate a metric per segment, such as revenue per product group

Validates an aggregated metric, for every segment defined by a segmentation field

Useful variants:

• Some validations benefit from other metrics than Sum, for example Mean.
• You can segment on multiple fields, e.g. product_group and country.

---

Count NULLs

Counts records where a field value is NULL

Useful variants:

• Track percentage of NULLs, by setting Metric to Percentage instead of Count.
• You can modify the filter to count something other than NULLs, for example count records where a field value is TRUE, 0 or "".
• Apply a Segmentation to track data per segment, for example, per ad campaign source.
• Use a Fixed threshold Equal 0 if no NULLs are accepted.

---

Validate uniqueness, e.g. unique primary keys

Counts duplicates and ensures that value is always 0

Useful variants:

• Pick multiple fields is useful, for example, when the primary key is a composition of multiple fields.

---

Validate a calculated ratio, such as conversion rate

Calculates and validates a ratio between two fields, for example sales per lead

Useful variants:

• Some validations benefit from other metrics than Sum ratio, for example Mean ratio.
• Apply a Segmentation to track data per segment, for example, per customer group.

---

Validate a time difference, such as mean session duration

Calculates and validates the difference between to timestamp fields

Useful variants:

• Some validations benefit from other metrics than Mean difference, for example Maximum difference.
• Apply a Segmentation to track data per segment, for example, per customer group.