Skip to main content
The Google Cloud Pub/Sub integration enables real-time data quality monitoring on streaming data in Validio. To integrate with Pub/Sub, you will need to add a credential and add a source for the data stream.
LimitationsThis integration has the following limitations:
  • Backfilling—Not supported because Pub/Sub does not support backfilling of historical data.
  • Data sequence—Pub/Sub requires a specific configuration to deliver messages in the correct order. If the message order is not configured properly, it will also impact how Validio receives the data. For more information, see Order messages in Google Cloud Pub/Sub documentation.

​
Prerequisites for integrating with Pub/Sub

For Validio to validate your Pub/Sub data stream requires:
  • Create a service account in Google Cloud console with the following access and permissions to the specific Pub/Sub stream:
  • Obtain a service account key in JSON file format for your service account. For more information, see Create and delete service account keys in Google Cloud documentation.

​
Add a Google Cloud credential

To add a Google Cloud credential for Pub/Sub,
  1. Navigate to Credentials and click + New credential.
  2. Under Namespace, select a namespace where the resources will be created.
  3. For Credential Type, select Google Cloud Platform (GCP).
  4. Fill in the Configuration parameter fields. Refer to the Google Cloud Credential Parameters table.
  5. (Optional) Click Test credential to validate that Validio can successfully access the Google Cloud account. If validation fails, check that you provided the correct parameter values.
  6. (Optional) Check Use for catalog and schema checks to add the BigQuery tables that this credential can access to the catalog. Pub/Sub topics and subscriptions are not added to the catalog.
  7. Click Create.
Once the credential is created, Validio will automatically start fetching data. You can add a source to monitor Pub/Sub data.

​
Google Cloud credential parameters

​
Add a Pub/Sub source

To add a source for Pub/Sub,
  1. Navigate to Sources and click + New source.
  2. Under Source type, select Google Cloud Pub/Sub.
  3. Fill in the configuration parameters. For more information, see Pub/Sub Configuration Parameters.
  4. Click Continue to create the source.

​
Pub/Sub configuration parameters