Overview
Destinations allow egress of bad data into a specified data source. This means that identified anomalies can be stored in a separate table or bucket.
With a Destination connector, you can analyze the anomalies at a later stage, in a tool of your choice. You can also integrate the anomaly table into your data pipelines with automatic jobs, for example, to quarantine bad datapoints.
Supported Validators
Destination connectors are currently only supported for the Numeric anomaly and Volume Validator types.
Validio supports a range of destination integrations with different cloud providers, Data Warehouses, and Data streams.
Missing Destinations?
We want to make Validio the world’s best Deep Data Observability platform, and would love your feedback. Let us know at [email protected].
Egress datapoints
All egress tables from Validio follow the same schema, regardless of which Source the data is validated from.
Only identified anomalies are written out to your destination.
Egress schema
| Field | Description |
|---|---|
| _validio_uuid | Auto generated unique identifier of the anomaly. |
| _validio_time | Timestamp of when the anomaly was caught. |
| _validio_is_anomaly | True for all egressed points. |
| _validio_applied_filters | Which filter(s) are applied and whether the filter labeled the datapoint an anomaly. |
| _validio_datapoint | The specific datapoint in questions containing all the field values. |
'_validio_applied_filters' details
One datapoint can be evaluated by multiple filters, such as on different data fields. As long as one of the filters labels the datapoint as an anomaly, the point is egressed.
The JSON egressed in the _validio_applied_filters lists all applied filters along with a boolean value that corresponds to whether the filter evaluates the datapoint as an anomaly or not.
You can find the validator_id in the URL when navigating a specific Validator in the platform.
Example
[
{
"validator_id": "MTR_ksleKxovmSG4peLApoQK51w",
"is_anomaly": true
},
{
"validator_id": "MTR_BLCpoSNzGHemp5T32089KhX",
"is_anomaly": false
}
]
Updated 27 days ago