HomeDocumentationChangelog
HomeRequest DemoContact
HomeRequest DemoContact

BigQuery

Create a BigQuery Source

Prepare credentials and permission in BigQuery

Certain credentials and permission are required for Validio to validate your data.

For detailed information about permissions in BigQuery, refer to Introduction to IAM.

Create a service account and assign roles

  1. Create a new service account for Validio in the project which contains the data you want to monitor. Alternatively, you can use an existing service account.
  2. Assign the following roles to the service account:
    1. BigQuery Data Viewer - We use this to read data and metadata.
    2. BigQuery Job User - We use this to query data.
    3. BigQuery Resource Viewer - We use this for various functionality and optimizations, e.g. by reading query history.
  3. If you want to monitor more projects:
    1. Grant the Validio service account access to each project, by specifying the Service account ID In the project's IAM-settings.
    2. Repeat step 2 for each project.

Create the Source in Validio

Credential parameters

FieldRequiredDescriptionExample
NameIdentifier for the credentials. Used when accessing Sources.service_acount_product_staging
Service accountUpload the JSON file, or paste the content of the JSON file containing the service account key.
For details, refer to Create and delete service account keys.

Configuration parameters

FieldRequiredDescriptionExample
NameIdentifier for the Source. Used when setting up validators.East_coast_weather_forecast
Project idName of the BigQuery project.weather-forecast
DatasetName of the dataset that contains the table.east-coast
TableName of the table to read data from.train-data
Cron presetDetermines how often to query the bucket for new data based on a preset option.

Select custom to use your own cron expression.
Cron expressionDetermines how often to query the table for new data based on cron expression.

Expression of cron presets are displayed here.

Used to enter your own cron expression.

Remove credentials and permission

If you want to remove credentials and permissions used for Validio, perform the following steps:

  1. If you created a separate Validio-project, remove it from BigQuery.
  2. Remove the service account used for Validio.

Note: Before you remove the service account, verify it isn't used elsewhere.