Pre-requisites

You will need:

  • A service account with permissions to access the specified GCS bucket
  • A base64 encoded service account key in JSON format

Service account

It is recommended you create a service account with read-only access for the Validio platform to ingest data from your GCS bucket. Details of these permissions can be found on GCP here.

The following permissions are required:

  • storage.buckets.get
  • storage.objects.get
  • storage.objects.list

Service account key

  • Obtain a service account key in JSON form for the service account. GCP instructions can be found here
  • Encode the service account key in base64

GCS configuration parameters

Field Required Description
Name Identifier for the connector. Used when setting up pipelines.
Bucket Name of the GCS bucket to ingest data from.
Folder Name of the folder in the bucket to ingest data from.
Polling interval value Determines how often to query the bucket for new data.
Unit The time unit used for the polling interval. E.g. a polling interval value of 2 with a unit of “hours” will poll the GCS bucket every two hours.
Credentials Base64 encoded form of the service account key in the JSON format.