Connectors

Sources

Authentication to read source and define features to ingest

Connector to data source enabling ingestion of data to be monitored and validated

Destinations

Authentication to write datapoint errors to a destination

Connector to data destination enabling real-time egress of data errors caught in datapoint pipelines

Pipelines

Dataset pipelines

Define batch logic, partitions and apply notification rules

Configuration of how a batch dataset should be defined, governing how dataset metrics are calculated.
Example: defining the dataset a mean should be calculated on

Define partitions, enabling metrics to be tracked on partitions of datasets.
Example: divide a dataset into partitions based on categorical ‘country’ feature, enabling e.g. mean calculation on ‘price’ feature by each country partition

Apply notification rules to a pipeline (created separately in the Notification wizard, see below)

Datapoint pipelines

Define ingestion logic, partitions and apply notification rules

Configure data pipeline ingestion logic - refer to the datapoint page for details

Define partitions, enabling metrics to be tracked on partitions of datasets

Apply notification rules to a pipeline (created separately in Notification wizard)

Notifications

Notifications

Define rule and channel for notification of alerts

Define notification rules of how alerts are grouped before sending through the alerts to the channel. Exists to mitigate alert fatigue

Configure and set-up notification channels

Monitors and filters

Monitors

Define and track dataset metrics on a specific feature

A monitor calculates and tracks a specific dataset metric
Example: mean of ‘age’ feature

Monitors are only applicable for Dataset pipelines

Multiple monitors can be configured for one dataset pipeline, allowing for multiple metrics to be calculated and tracked

Filters

Define filters evaluating individual datapoints on a specific feature

Evaluate individual data points based on specified filter logic
Example: filter all datapoints with value between 0-18 in ‘age feature’

Filters are only applicable for datapoint pipelines

Multiple filters can be configured for one datapoint pipeline, allowing for multiple different filters

Alerts

Alerts

Define logic for what should be considered erroneous data and be alerted for

Monitors: defines when a calculated metric a monitor tracks should be defined as erroneous data and alerted for
Example: alert when min value of ‘price’ feature in a dataset is under 0

Filters: define logic of when datapoints passing (or not passing) a filter should be alerted for
Example: alert when more than 10 datapoints ingested in the last micro-batch did not pass the filter

All alerts are visible in the UI, and will be grouped according to notification rules before sending it through the notification channel