Pub/Sub Lite

To integrate with Google Cloud Pub/Sub Lite, you will need to add a credential and add a source for the data stream.

πŸ“˜

Note

This integration has the following limitations:

  • Backfilling--Because Pub/Sub Lite does not support backfilling of historic data, this feature may not work as expected in Validio.
  • Data sequence--Pub/Sub Lite requires a specific configuration to deliver messages in the correct order. If the message order is not configured properly, it will also impact how Validio receives the data. For more information, see Order messages in Google Cloud Pub/Sub documentation.

Prerequisites for Integrating with Pub/Sub Lite

For Validio to validate your Pub/Sub data stream requires:

  • Create a service account in Google Cloud console with the following access and permissions to the specific Pub/Sub stream:
  • Obtain a service account key in JSON file format for your service account. For more information, see Create and delete service account keys in Google Cloud documentation.

Add a Pub/Sub Lite Credential

To add a Pub/Sub Lite credential,

  1. Navigate to Observability > Credentials and click + New Credential.
  2. Under Credential Type, select GCP Credential.
  3. Fill in the configuration parameters for the GCP credential. For more information, see GCP Credential Parameters.
  4. Click Create.

GCP Credential Parameters

FieldRequiredDescription
Nameβœ…Identifier for the credentials. Used when accessing Sources.
Service accountβœ…Upload the JSON file containing the service account key, or
paste the content of the JSON file containing the service account key.

Add a Pub/Sub Lite Source

To add a source for Pub/Sub Lite,

  1. Navigate to Observability > Sources and click + New Source.
  2. Under Source type, select Google Cloud Pub/Sub Lite.
  3. Fill in the configuration parameters. For more information, see Pub/Sub Configuration Parameters.
  4. Click Continue to create the source.

Pub/Sub Lite Configuration Parameters

FieldRequiredDescription
Nameβœ…Identifier for the Source. Used when setting up Validators.
Project idβœ…Id of the Pub/Sub Lite project to read data from.
Locationβœ…Location of the Pub/Sub Lite stream to read from.
Subscription idβœ…Id of the Pub/Sub Lite stream to read data from.
Message formatβœ…JSON, AVRO or PROTOBUF
Message schemaβœ…JSON - Not needed, Validio automatically infers schema
AVRO - Refer to Avro Schema Declaration.
PROTOBUF - Refer to Protobuf Message Type.