Prerequisites for Google BigQuery

Certain credentials and permission are required for the service account in Google Cloud Console for Validio to validate your data. For detailed information about permissions in BigQuery, refer to Introduction to IAM.

Create a Service Account and Assign Roles

1. Create a new service account for Validio in the project which contains the data you want to monitor. Alternatively, you can use an existing service account.
2. Assign the following roles to the service account, at the project-level:
   1. BigQuery Data Viewer - to read data and metadata.
   2. BigQuery Job User - to query data.
   3. BigQuery Resource Viewer - for various functionality and optimizations, for example, by reading query history.
   4. Metadata Viewer - to read metadata, specifically for the metadata validators.
3. Ensure Cloud Resource Manager API is enabled on the project. For more information, refer to API Console Help.
4. Server Usage Consumer - (Optional) If you want to use a different GCP project for billing purposes, this role must be granted on the service account.
5. If you want to monitor more projects:
   1. Grant the Validio service account access to each project, by specifying the Service account ID In the project's IAM-settings.https://docs.validio.io/v5.0/docs/gcs#/
   2. Repeat step 2 and 3 for each project.

Add a Google Cloud Credential

To add a Google Cloud credential for BigQuery,

1. Navigate to Credentials and click + New Credential.
2. Under Namespace, select a namespace where the resources will be created.
3. For Credential Type, select GCP Credential.
4. Fill in the credential parameter fields. Refer to the Google Cloud Credential Parameters table.
5. Check Use for catalog to automatically discover credentials and add them to the catalog page.
6. Click Create.

Validio will validate the connection to the Google Cloud account. If validation passes, Validio will automatically start fetching data. If validation fails, check that you provided the correct parameter values and try again.

Once the credential is created, you can add a source to monitor BigQuery data.

Google Cloud Credential Parameters

Add a BigQuery Source

To add a source for Google BigQuery,

1. Navigate to Sources and click + New source.
2. Under Source type, click Data Warehouse to select Google BigQuery.
3. Under Config,
   1. Select the valid Credential or create a new credential to authenticate your connection to the data warehouse.
   2. Enter the Project ID, Dataset, and Table to specify where the data comes from. Selecting more than one table will create a new source for each table. Refer to the Configuration Parameters table.
   3. Set how many days of Historic data to use when you start the source.
   4. Set the Polling schedule, which is how frequently the validators on the source will check for changes.
4. Under Schema, click Continue to automatically infer the schema fields from the tables you selected. If you select many tables, this operation can take a few minutes to complete.
5. Under Source details,
   1. Add Tags to help group related sources or to use for routing notifications.
   2. Add an Owner who will be the contact for incident notifications.
6. (Optional) Under Validators, check each of the recommended metadata validators that you want to add to your source.
7. Click Continue to create the source.
   Source names are generated automatically and will be displayed when the source creation completes. If there are more than 5 sources, you will see the names for the first five and a count of the remaining sources.

Configuration parameters

Remove credentials and permission

If you want to remove credentials and permissions used for Validio, perform the following steps:

1. If you created a separate Validio-project, remove it from BigQuery.
2. Remove the service account used for Validio.

📘

Note

Before removing the service account, verify that it is not used elsewhere.