Concepts and Terminology

This page contains an overview of the key conceptual parts used across the Validio platform.

Key concepts model.
  1. Credentials are used to access your Source. One set of Credentials can be used to set up multiple Sources.

  2. A Source is a connector to one source system, such as Data Warehouse, Data Stream, or an Object Storage. A Source is defined as one table in a Data Warehouse, one topic in a Data Stream, or one specified set of schema-conformant files in an Object Storage.
    Segmentations, Windows, and Validators are defined for each Source

  3. Validio validates data using metrics calculated over a subset of the data in a Source. Each such subset is called a Window. A Window can be defined as a time interval, a fixed sized batch, or a file. There is also a Global Window, which considers all data in the Source.
    Each Source must have at least one window, but several windows can be created for each Source.

  4. Segmentation allows validation per segment, also referred to as group. You can think of this as a GROUP BY statement in SQL. Each Source has at least one segment. The default Segmentation is called Unsegmented.

  5. A Validator specifies what metrics to validate on what fields and what Threshold should be considered acceptable. For Validators you can also select a defined Segmentation and Window, and configure filters.
    Each Source can have one or more Validators.

  6. Optionally, a Notification rule can be used to send incidents to specified channels, such as Slack. Each notification rule can include incidents from multiple Sources.

  7. Each Notification rule has a notification Channel attached. The same Channel can be used for multiple Notification rules.

  8. Lineage describes how data flows through a data stack, from its origin to its final use. For some source types Lineage is created automatically, based on a Credential, or a dbt Manifest file. For others, Lineage can be created manually, based on a Source.