Kafka

The Apache Kafka integration enables real-time data quality monitoring on streaming data in Validio. To integrate with Kafka, you will need to add a credential and add a source for the data stream.

​

Prerequisites for integrating with Kafka

Certain credentials and permission are required for Validio to validate your data:

• A service account with access and permissions to the specific Kafka clusters. For more information, refer to Kafka security documentation and Confluent: Kafka SASL/PLAIN authentication.

​

Add a Kafka credential

To add a Kafka credential in Validio,

1. Navigate to Credentials and click + New credential.
2. Under Namespace, select a namespace where the resources will be created.
3. For Credential Type, select either Kafka SSL or Kafka SASL SSL Plain, depending on the authentication method for your Kafka clusters.
4. Fill in the Configuration parameter fields. Refer to the Kafka Credential Parameters section.
5. (Optional) Click Test credential to validate that Validio can successfully access the Kafka cluster. If validation fails, check that you provided the correct parameter values.
6. Check Use for catalog and schema checks to automatically discover credentials and add them to the catalog page.
7. Click Create.

​

Kafka credential parameters

The Kafka credential configuration depends on the authentication method for your Kafka clusters, either Kafka SSL or Kafka SASL SSL Plain.

​

Kafka SSL parameters

The Kafka SSL Credentials credential type requires the following parameters and certificates:

​

Kafka SASL SSL Plain parameters

The Kafka SASL Plain Credential credential type, which uses TLS/SSL as transport layer, requires the following parameters and certificates:

​

Add a Kafka source

To add a source for Kafka,

1. Navigate to Sources and click + New source.
2. Under Source type, select Apache Kafka.
3. Fill in the configuration parameters. For more information, see Kafka Configuration Parameters.
4. Click Continue to create the source.

​

Kafka configuration parameters