> ## Documentation Index
> Fetch the complete documentation index at: https://docs.validio.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Pub/Sub

> Connect Validio to Google Cloud Pub/Sub for real-time data quality monitoring on streaming data.

The Google Cloud Pub/Sub integration enables real-time data quality monitoring on streaming data in Validio. To integrate with Pub/Sub, you will need to add a credential and add a source for the data stream.

> [!NOTE]
>
> **Limitations**
>
> This integration has the following limitations:
>
> * **Backfilling**--Not supported because Pub/Sub does not support backfilling of historical data.
> * **Data sequence**--Pub/Sub requires a specific configuration to deliver messages in the correct order. If the message order is not configured properly, it will also impact how Validio receives the data. For more information, see [Order messages](https://cloud.google.com/pubsub/docs/ordering) in Google Cloud Pub/Sub documentation.

## Prerequisites for integrating with Pub/Sub

For Validio to validate your Pub/Sub data stream requires:

* Create a service account in Google Cloud console with the following access and permissions to the specific Pub/Sub stream:
  * `roles/pubsub.subscriber` to consume messages from a subscription.
  * `roles/pubsub.viewer` to view topics and subscriptions.
  * For more information, see [Access Control with IAM for Pub/Sub](https://cloud.google.com/pubsub/docs/access-control) and [Authenticate to Pub/Sub](https://cloud.google.com/pubsub/docs/authentication) in Google Cloud documentation.
* Obtain a service account key in JSON file format for your service account. For more information, see [Create and delete service account keys](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) in Google Cloud documentation.

## Add a Google Cloud credential

To add a Google Cloud credential for Pub/Sub,

1. Navigate to **Credentials** and click **+ New credential**.
2. Under **Namespace**, select a namespace where the resources will be created.
3. For **Credential Type**, select **Google Cloud Platform (GCP)**.
4. Fill in the **Configuration** parameter fields. Refer to the [Google Cloud Credential Parameters](#google-cloud-credential-parameters) table.
5. (Optional) Click **Test credential** to validate that Validio can successfully access the Google Cloud account. If validation fails, check that you provided the correct parameter values.
6. (Optional) Check **Use for catalog and schema checks** to add the BigQuery tables that this credential can access to the [catalog](https://docs.validio.io/docs/catalog/catalog-assets.md). Pub/Sub topics and subscriptions are not added to the catalog.
7. Click **Create**.

Once the credential is created, Validio will automatically start fetching data. You can add a source to monitor Pub/Sub data.

### Google Cloud credential parameters

| Field | Description |
| :- | :- |
| Name | Identifier for the credentials. Used when accessing Sources. |
| Billing project ID | (Optional) Specify the project to use for quota and billing purposes, if it is different from the resource project. Note: The service account must have the `Server Usage Consumer` role. |
| Service account JSON-file | Upload the JSON file, or paste the content of the JSON file containing the service account key. For details, refer to [Create and delete service account keys](https://cloud.google.com/iam/docs/creating-managing-service-account-keys). |

## Add a Pub/Sub source

To add a source for Pub/Sub,

1. Navigate to **Sources** and click **+ New source**.
2. Under **Source type,** select **Google Cloud Pub/Sub**.
3. Fill in the configuration parameters. For more information, see [Pub/Sub Configuration Parameters](#pub/sub-configuration-parameters).
4. Click **Continue** to create the source.

### Pub/Sub configuration parameters

| Field | Description |
| :- | :- |
| Name | Identifier for the Source. Used when setting up Validators. |
| Project Id | Id of the Pub/Sub project to read data from. |
| Subscription Id | Id of the Pub/Sub stream to read data from. |
| Message format | JSON, AVRO or PROTOBUF |
| Message schema | For JSON, Validio automatically infers the schema. For AVRO and PROTOBUF, you can upload schemas. For more information, see [Data Stream Sources](https://docs.validio.io/docs/integrations/data-streams.md). |