Prerequisite credentials
You need the following to configure S3 credentials in the Validio platform:
- A IAM user with permissions to access the specified S3 bucket
- Credentials for the IAM user
Permissions for IAM user
Certain permissions and roles are required for the IAM user.
For more information, refer to Actions, resources, and condition keys for Amazon S3.
IAM user for Source config
We recommended that you create an IAM user with read access for the Validio platform to ingest data from your S3 bucket.
The following permissions must be assigned to your IAM user:
s3.ListBuckets3.GetObject
Access key
You must supply an Access key and secret key to authenticate to the S3 bucket.
For more information, refer to Managing access keys for IAM users.
Credential parameters
| Field | Required | Description | Example |
|---|---|---|---|
| Name | ✅ | Identifier for the credentials. Used when accessing sources. | service_acount_product_staging |
| Access key | ✅ | Access key with permissions to access the S3 bucket. | |
| Secret key | ✅ | Secret key for the specified access key. |
Configuration parameters
| Field | Required | Description | Example |
|---|---|---|---|
| Name | ✅ | Identifier for the connector. Used when setting up Validators. | |
| Bucket | ✅ | Name of the S3 bucket to ingest data from. | |
| Prefix | ✅ | Prefix of the folder in the S3 bucket to ingest data from. | If “/a/b” is specified as a prefix, Validio only reads the files that are located in the “/a/b” folder |
| File pattern | Filter what files to ingest, based on file names and regex expressions. | file_chunk_number_\d+.csv | |
| Cron preset | Determines how often to query the bucket for new data based on a preset option. Select custom to use your own cron expression. | ||
| Cron expression | ✅ | Determines how often to query the bucket for new data based on cron expression. Expression of cron presets are displayed here. Used to enter your own cron expression. | |
| CSV parser config | ✅ | Specify which delimiter is used in the CSV file. Optionally, specify a character or string used to represent Null. | CSV delimiter: , Null marker: NULL |
Updated 4 days ago