Pub/Sub
To integrate with Google Cloud Pub/Sub, you will need to add a credential and add a source for the data stream.
Limitations
This integration has the following limitations:
- Backfilling--Not supported because Pub/Sub does not support backfilling of historical data.
- Data sequence--Pub/Sub requires a specific configuration to deliver messages in the correct order. If the message order is not configured properly, it will also impact how Validio receives the data. For more information, see Order messages in Google Cloud Pub/Sub documentation.
Prerequisites for Integrating with Pub/Sub
For Validio to validate your Pub/Sub data stream requires:
- Create a service account in Google Cloud console with the following access and permissions to the specific Pub/Sub stream:
roles/pubsub.subscriberto consume messages from a subscription.roles/pubsub.viewerto view topics and subscriptions.- For more information, see Access Control with IAM for Pub/Sub and Authenticate to Pub/Sub in Google Cloud documentation.
- Obtain a service account key in JSON file format for your service account. For more information, see Create and delete service account keys in Google Cloud documentation.
Add a Google Cloud Credential
To add a Google Cloud credential for Pub/Sub,
- Navigate to Credentials and click + New Credential.
- Under Namespace, select a namespace where the resources will be created.
- For Credential Type, select GCP Credential.
- Fill in the credential parameter fields. Refer to the Google Cloud Credential Parameters table.
- Check Use for catalog to automatically discover credentials and add them to the catalog page.
- Click Create.
Validio will validate the connection to the Google Cloud account. If validation passes, Validio will automatically start fetching data. If validation fails, check that you provided the correct parameter values and try again.
Once the credential is created, you can add a source to monitor Pub/Sub data.
Google Cloud Credential Parameters
Field | Description |
|---|---|
Name | Identifier for the credentials. Used when accessing Sources. |
Billing project ID | (Optional) Specify the project to use for quota and billing purposes, if it is different from the resource project. Note: The service account must have the |
Service account JSON-file | Upload the JSON file, or paste the content of the JSON file containing the service account key. For details, refer to Create and delete service account keys. |
Add a Pub/Sub Source
To add a source for Pub/Sub,
- Navigate to Observability > Sources and click + New Source.
- Under Source type, select Google Cloud Pub/Sub.
- Fill in the configuration parameters. For more information, see Pub/Sub Configuration Parameters.
- Click Continue to create the source.
Pub/Sub Configuration Parameters
Field | Description |
|---|---|
Name | Identifier for the Source. Used when setting up Validators. |
Project Id | Id of the Pub/Sub project to read data from. |
Subscription Id | Id of the Pub/Sub stream to read data from. |
Message format | JSON, AVRO or PROTOBUF |
Message schema | For JSON, Validio automatically infers the schema. For AVRO and PROTOBUF, you can upload schemas. For more information, see Data Stream Sources . |
Updated 13 days ago