The Apache Kafka integration enables real-time data quality monitoring on streaming data in Validio. To integrate with Kafka, you will need to add a credential and add a source for the data stream.

Prerequisites for Integrating with Kafka

Certain credentials and permission are required for Validio to validate your data:

A service account with access and permissions to the specific Kafka clusters. For more information, refer to Kafka security documentation and Confluent: Kafka SASL/PLAIN authentication.

Add a Kafka Credential

To add a Kafka credential in Validio,

Navigate to Credentials and click + New credential.
Under Namespace, select a namespace where the resources will be created.
For Credential Type, select either Kafka SSL or Kafka SASL SSL Plain, depending on the authentication method for your Kafka clusters.
Fill in the Configuration parameter fields. Refer to the Kafka Credential Parameters section.
(Optional) Click Test credential to validate that Validio can successfully access the Kafka cluster. If validation fails, check that you provided the correct parameter values.
Check Use for catalog and schema checks to automatically discover credentials and add them to the catalog page.
Click Create.

Kafka Credential Parameters

The Kafka credential configuration depends on the authentication method for your Kafka clusters, either Kafka SSL or Kafka SASL SSL Plain.

Kafka SSL Parameters

The Kafka SSL Credentials credential type requires the following parameters and certificates:

Field	Description
Name	Identifier for the credentials. Used when accessing Sources.
Bootstrap server	The address of Kafka load balancer or Kafka cluster address. Use a comma to separate multiple addresses.
CA certificate	A certificate of Certificate authority (CA) in CRT format.
Client certificate	Client SSL certificate in PEM format.
Client private key	Private keys of client certificate in PEM format.
Client private key password	Password or passphrase of private keys for the client certificate.

You can either provide the certificates and client private key as plain text or upload the files.

Kafka SASL SSL Plain Parameters

The Kafka SASL Plain Credential credential type, which uses TLS/SSL as transport layer, requires the following parameters and certificates:

Field	Description
Name	Identifier for the credentials. Used when accessing Sources.
Bootstrap server	The address of Kafka load balancer or Kafka cluster address. Use a comma to separate multiple addresses.
Username	Username for the SASL Plain Credential.
Password	Password for the SASL Plain Credential.

Add a Kafka Source

To add a source for Kafka,

Navigate to Sources and click + New source.
Under Source type, select Apache Kafka.
Fill in the configuration parameters. For more information, see Kafka Configuration Parameters.
Click Continue to create the source.

Kafka Configuration Parameters

Field	Description
Name	Identifier for the Source. Used when setting up Validators.
Cluster	Name of the Kafka cluster to read data from.
Message format	JSON, AVRO, or PROTOBUF
Message schema	For JSON, Validio automatically infers the schema. For AVRO and PROTOBUF, you can upload schemas. For more information, see Data Stream Sources.