Prerequisite credentials

You need the following to configure S3 credentials in the Validio platform:

  • A IAM user with permissions to access the specified S3 bucket
  • Credentials for the IAM user

πŸ“˜

Permissions for IAM user

Certain permissions and roles are required for the IAM user.
For more information, refer to Actions, resources, and condition keys for Amazon S3.

IAM user for Source config

We recommended that you create an IAM user with read access for the Validio platform to ingest data from your S3 bucket.

The following permissions must be assigned to your IAM user:

  • s3.ListBucket
  • s3.GetObject

Access key

You must supply an Access key and secret key to authenticate to the S3 bucket.
For more information, refer to Managing access keys for IAM users.

Credential parameters

FieldRequiredDescriptionExample
Nameβœ…Identifier for the credentials. Used when accessing sources.service_acount_product_staging
Access keyβœ…Access key with permissions to access the S3 bucket.
Secret keyβœ…Secret key for the specified access key.

Configuration parameters

FieldRequiredDescriptionExample
Nameβœ…Identifier for the connector. Used when setting up Validators.
Bucketβœ…Name of the S3 bucket to ingest data from.
Prefixβœ…Prefix of the folder in the S3 bucket to ingest data from.If β€œ/a/b” is specified as a prefix, Validio only reads the files that are located in the β€œ/a/b” folder
File patternFilter what files to ingest, based on file names and regex expressions.file_chunk_number_\d+.csv
Cron presetDetermines how often to query the bucket for new data based on a preset option.

Select custom to use your own cron expression.
Cron expressionβœ…Determines how often to query the bucket for new data based on cron expression.

Expression of cron presets are displayed here.

Used to enter your own cron expression.
CSV parser configβœ…Specify which delimiter is used in the CSV file.

Optionally, specify a character or string used to represent Null.
CSV delimiter: ,

Null marker: NULL