HomeDemoContact

Overview

Destinations allow egress of bad data into a specified data source. This means that identified anomalies can be stored in a separate table or bucket.

With a Destination connector, you can analyze the anomalies at a later stage, in a tool of your choice. You can also integrate the anomaly table into your data pipelines with automatic jobs, for example, to quarantine bad datapoints.

πŸ“˜

Supported Validators

Destination connectors are currently only supported for the Numeric anomaly and Volume Validator types.

Validio supports a range of destination integrations with different cloud providers, Data Warehouses, and Data streams.

Destination systemDestination type
Data warehouseBigQuery
Data warehouseSnowflake
Data streamKinesis

πŸ“˜

Missing Destinations?

We want to make Validio the world’s best Deep Data Observability platform, and would love your feedback. Let us know at [email protected].

Egress datapoints

All egress tables from Validio follow the same schema, regardless of which Source the data is validated from.

πŸ“˜

Only identified anomalies are written out to your destination.

Egress schema

FieldDescription
_validio_uuidAuto generated unique identifier of the anomaly.
_validio_timeTimestamp of when the anomaly was caught.
_validio_is_anomalyTrue for all egressed points.
_validio_applied_filtersWhich filter(s) are applied and whether the filter labeled the datapoint an anomaly.
_validio_datapointThe specific datapoint in questions containing all the field values.

'_validio_applied_filters' details

One datapoint can be evaluated by multiple filters, such as on different data fields. As long as one of the filters labels the datapoint as an anomaly, the point is egressed.

The JSON egressed in the _validio_applied_filters lists all applied filters along with a boolean value that corresponds to whether the filter evaluates the datapoint as an anomaly or not.

You can find the validator_id in the URL when navigating a specific Validator in the platform.

Example

[
  {
    "validator_id": "MTR_ksleKxovmSG4peLApoQK51w",
    "is_anomaly": true
  },
  {
    "validator_id": "MTR_BLCpoSNzGHemp5T32089KhX",
    "is_anomaly": false
  }
]