> ## Documentation Index
> Fetch the complete documentation index at: https://docs.validio.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Google BigQuery

> Create a BigQuery Source

Connect Validio to Google BigQuery to monitor data quality across your BigQuery datasets and tables.

## Prerequisites for Google BigQuery

Certain [credentials](https://docs.validio.io/v11.0/docs/credentials.md) and permission are required for the service account in Google Cloud Console for Validio to validate your data. For detailed information about permissions in BigQuery, refer to [Introduction to IAM](https://cloud.google.com/bigquery/docs/access-control).

> [!WARNING]
>
> **Credential Permission Requirements**
>
> Validio Credentials **require** `VIEWER` access rights when connecting to sources to read and access data. Admins must ensure that they do not provide `EDITOR` access rights to their credentials.

### Create a service account and assign roles

1. Create a new service account for Validio in the project which contains the data you want to monitor. Alternatively, you can use an existing service account.
2. Assign the following roles to the service account, at the project-level:
   1. `BigQuery Data Viewer` - to read data and metadata.
   2. `BigQuery Job User` - to query data.
   3. `BigQuery Resource Viewer` -  for various functionality and optimizations, for example, by reading query history.
   4. `Metadata Viewer` - to read metadata, specifically for the metadata validators.
3. Ensure `Cloud Resource Manager API` is enabled on the project. For more information, refer to [API Console Help](https://support.google.com/googleapi/answer/6158841?hl=en).
4. `Server Usage Consumer` - (Optional) If you want to use a different GCP project for billing purposes, this role must be granted on the service account.
5. If you want to monitor more projects:
   1. Grant the Validio service account access to each project, by specifying the Service account ID In the project's IAM-settings.
   2. Repeat step 2 and 3 for each project.

## Add a Google Cloud credential

To add a Google Cloud credential for BigQuery,

1. Navigate to **Credentials** and click **+ New credential**.
2. Under **Namespace**, select a namespace where the resources will be created.
3. For **Credential Type**, select **GCP Credential**.
4. Fill in the **Configuration** parameter fields. Refer to the [Google Cloud Credential Parameters](#google-cloud-credential-parameters) table.
5. (Optional) Click **Test credential** to validate that Validio can successfully access the GCP account. If validation fails, check that you provided the correct parameter values.
6. (Optional) Check **Use for catalog and schema checks** to automatically discover assets from this credential and add them to the Catalog page.
7. Click **Create credential**.

Validio will automatically start fetching data and you will be able to view BigQuery assets and their relationships in the [Catalog](https://docs.validio.io/v11.0/docs/catalog.md) (if selected) and [Lineage](https://docs.validio.io/v11.0/docs/lineage.md) pages.

Once the credential is created, you can add a [source](https://docs.validio.io/v11.0/docs/sources.md) to monitor BigQuery data.

### Google Cloud credential parameters

| Field | Description |
| :- | :- |
| Name | Identifier for the credentials. Used when accessing Sources. |
| Billing project ID | (Optional) Specify the project to use for quota and billing purposes, if it is different from the resource project. Note: The service account must have the `Server Usage Consumer` role. |
| Service account JSON-file | Upload the JSON file, or paste the content of the JSON file containing the service account key. For details, refer to [Create and delete service account keys](https://cloud.google.com/iam/docs/creating-managing-service-account-keys). |

## Add a BigQuery source

To add a source for Google BigQuery,

1. Navigate to **Sources** and click **+ New source**.
2. Under **Source type**, click  **Data Warehouse** to select **Google BigQuery**.
3. Under **Config**,
   1. Select the valid **Credential** or create a new credential to authenticate your connection to the data warehouse.
   2. You have two options for specifying the dataset and tables to monitor:
      1. **Select an existing table** -- Enter the **Project ID**, **Dataset**, and **Table**to specify where the data comes from. Selecting more than one table will create a new source for each table. Refer to the [Source Configuration Parameters](#source-configuration-parameters) table.
      2. **Use Custom SQL** -- Write a valid SQL query to specify the tables to monitor.
   3. Set the **Polling schedule**, which is how frequently the validators on the source will check for changes.
4. Under **Schema**, click **Continue** to automatically infer the schema fields from the tables you selected. If you select many tables, this operation can take a few minutes to complete.
5. Under **Source details**,
   1. Add **Tags** to help group related sources or to use for routing notifications.
   2. Add an **Owner** who will be the contact for incident notifications.
   3. Assign a **Priority**, which indicates the importance of incidents detected on this source.
6. (Optional) Under **Validators**, check each of the recommended metadata validators that you want to add to your source.
7. Click **Continue** to create the source.
   Source names are generated automatically and  will be displayed when the source creation completes. If there are more than 5 sources, you will see the names for the first five and a count of the remaining sources.

### Source configuration parameters

| Field | Description | Example |
| :- | :- | :- |
| Project id | Name of the BigQuery project. | *weather-forecast* |
| Dataset | Name of the dataset that contains the table. | *east-coast* |
| Table | Name of the table to read data from. | *train-data* |

## Remove credentials and permission

If you want to remove credentials and permissions used for Validio, perform the following steps:

1. If you created a separate Validio-project, remove it from BigQuery.
2. Remove the service account used for Validio.

> [!NOTE]
>
> Before removing the service account, verify that it is not used elsewhere.