A circuit breaker is a pattern where your orchestration tool stops a data pipeline from continuing when Validio detects data quality incidents in freshly loaded data. By placing Validio between an ingestion step and the steps that depend on it, you prevent low-quality data from propagating to downstream tables, dashboards, and models.

Validio supports this pattern today through the Apache Airflow integration, the Validio SDK, and the Validio API, so you can gate any orchestrator.

How a Circuit Breaker Works

A circuit breaker wraps a data quality check around the point in your pipeline where new data lands. The pattern has three steps:

1. Poll the source. After an upstream job (such as an ETL load) writes new data, trigger a Validio source poll so its validators evaluate the freshly landed data immediately, instead of waiting for the next scheduled poll.
2. Check for incidents. Once the poll completes, check whether it produced any incidents for that source.
3. Break or continue. If the number or severity of incidents exceeds the threshold you allow, fail the task and stop the downstream steps. Otherwise, let the pipeline continue.

You decide how strict the breaker is — for example, allow a small number of low-severity incidents but break on any high-severity incident.

Circuit Breaking with Apache Airflow

The Validio for Airflow integration provides operators and sensors that implement this pattern natively in a DAG:

1. Use the ValidioPollSourceOperator to trigger a source poll after your data-loading task completes.
2. Use the ValidioIncidentsSensor to gate the DAG. The sensor pokes Validio for recent incidents and can be configured with an allowed number of incidents and allowed severities. It triggers — failing the gate — once there are more incidents outside the allowed severities than you permit, stopping the downstream tasks.

For installation, connection setup, and the full list of parameters for each operator and sensor, see Validio for Airflow.

Circuit Breaking with the SDK

If you use an orchestrator other than Airflow, the Validio SDK is the simplest way to build a circuit breaker. The SDK's poll_source helper triggers a manual poll, waits for it to complete, and reports whether the poll produced any new incidents — all in a single call:

1. Poll and check. Call poll_source with the source's id or resource name. It starts the poll, waits for it to finish, and then checks for incidents created since the poll started.
2. Break or continue. Inspect the result it returns. If the poll couldn't complete, or if it produced more new incidents than you allow, raise an error or return a non-zero exit code so your orchestrator halts the pipeline; otherwise let it proceed.

See the Manually Poll a Source and Its Validators recipe for a complete example.

Circuit Breaking with the API

If you can't use the Python SDK, you can build the same circuit breaker directly against the Validio API from any orchestrator that can call it:

1. Trigger and track a poll. Start a manual poll and wait for it to complete.
2. Check for incidents from that poll. Query the incidents API filtered by the source and by a createdAt time at or after the poll started, so you only count incidents that this poll produced.
3. Break or continue. Apply your threshold to the result. If it is exceeded, raise an error or return a non-zero exit code so your orchestrator halts the pipeline; otherwise allow it to proceed.

Related

• Validio for Airflow
• Validio SDK
• Validio API
• About Incidents
• About Notifications