HomeDocumentationRecipesChangelog
HomeRequest DemoContact
Documentation
HomeRequest DemoContact

Google BigQuery

Create a BigQuery Source

Suggest Edits

Prepare credentials and permission in Google Cloud console

Certain credentials and permission are required for Validio to validate your data. For detailed information about permissions in BigQuery, refer to Introduction to IAM.

Create a service account and assign roles

  1. Create a new service account for Validio in the project which contains the data you want to monitor. Alternatively, you can use an existing service account.
  2. Assign the following roles to the service account:
    1. BigQuery Data Viewer - We use this to read data and metadata.
    2. BigQuery Job User - We use this to query data.
    3. BigQuery Resource Viewer - We use this for various functionality and optimizations, for example, by reading query history.
  3. Ensure Cloud Resource Manager API is enabled on the project. For more information, refer to API Console Help.
  4. If you want to monitor more projects:
    1. Grant the Validio service account access to each project, by specifying the Service account ID In the project's IAM-settings.
    2. Repeat step 2 and 3 for each project.

Create the Source in Validio

Credential parameters

FieldRequiredDescriptionExample
Name✅Identifier for the credentials. Used when accessing Sources.service_acount_product_staging
Service account✅Upload the JSON file, or paste the content of the JSON file containing the service account key.
For details, refer to Create and delete service account keys.

Configuration parameters

FieldRequiredDescriptionExample
Name✅Identifier for the Source. Used when setting up validators.East_coast_weather_forecast
Project id✅Name of the BigQuery project.weather-forecast
Dataset✅Name of the dataset that contains the table.east-coast
Table✅Name of the table to read data from.train-data
Cron presetDetermines how often to query the bucket for new data based on a preset option.

Select custom to use your own cron expression.
Cron expression✅Determines how often to query the table for new data based on cron expression. The polling schedule for cron-expressions uses UTC timezone settings.

Displays the expression of cron presets. If you selected a custom preset, you can enter the cron expression.

For help writing cron expressions, refer to Crontab Guru.

Remove credentials and permission

If you want to remove credentials and permissions used for Validio, perform the following steps:

  1. If you created a separate Validio-project, remove it from BigQuery.
  2. Remove the service account used for Validio.

📘

Note

Before removing the service account, verify that it is not used elsewhere.

Updated 5 months ago