Google Cloud Storage (GCS)
Prepare credentials and permission in Google Cloud console
Certain credentials and permission are required for Validio to validate your data:
- A service account with access and permissions to the specified GCS bucket.
- A JSON file containing the service account key.
Permissions for service accounts
Certain permissions and roles are required for the service account.
For more information, refer to GCP - Introduction to IAM.
Service account for Source config
We recommend that you create a service account where the Validio platform is granted access to the GCS bucket you want to read data from.
The following roles must be assigned to the service account:
storage.buckets.getstorage.objects.getstorage.objects.list
Service account key
- Obtain a service account key in JSON file format for your service account.
For details, refer to Create and delete service account keys. - Provide the service account to the Credentials field in Validio by either:
- Upload the JSON file to the
JSON filefield. - Paste the content of the JSON file into the
JSON textfield.
- Upload the JSON file to the
Credential parameters
| Field | Required | Description | Example |
|---|---|---|---|
| Name | ✅ | Identifier for the credentials. Used when accessing Sources. | service_account_product_staging |
| Service account | ✅ | Upload the JSON file or paste the content of the JSON file containing the service account key. |
Configuration parameters
| Field | Required | Description | Example |
|---|---|---|---|
| Name | ✅ | Identifier for the Source. Used when setting up validators. | |
| Project Id | ✅ | Identifier of the GCS project. | |
| Bucket | ✅ | Name of the GCS bucket that contains the folder. | |
| Folder | ✅ | Name of the folder to read data from. | |
| File pattern | Filter what files to read, based on file names and regex expressions. | file_chunk_number_\d+.csv | |
| Cron preset | Determines how often to query the bucket for new data based on a preset option. Select custom to use your own cron expression. | ||
| Cron expression | ✅ | Determines how often to query the bucket for new data based on cron expression. Expression of cron presets are displayed here. Used to enter your own cron expression. | |
| File format | ✅ | Select the type of file: CSV, Parquet, or JSON. For CSV file formats, specify the delimiter used in the CSV file and (optional) the character or string used to represent a null value. | CSV delimiter: , Null marker: NULL |
Updated 10 months ago