Amazon S3
Prepare credentials and permission in AWS Management Console
Certain credentials and permission are required for Validio to validate your data:
- A IAM user with access and permissions to the specified S3 bucket.
- Credentials for the IAM user.
Permissions for IAM user
Certain permissions and roles are required for the IAM user.
For more information, refer to Actions, resources, and condition keys for Amazon S3.
IAM user for Source config
We recommend that you create an IAM user where the Validio platform is granted access to the S3 bucket you want to read data from.
The following permissions must be assigned to your IAM user:
s3:ListBuckets3:GetObject
Access key
You must supply an Access key and secret key to authenticate to the S3 bucket.
For more information, refer to Managing access keys for IAM users.
Credential parameters
| Field | Required | Description | Example |
|---|---|---|---|
| Name | ✅ | Identifier for the credentials. Used when accessing Sources. | service_acount_product_staging |
| Access key | ✅ | Access key with permissions to access the S3 bucket. | AKIAIOSFODNN7EXAMPLE |
| Secret key | ✅ | Secret key for the specified access key. | wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY |
Configuration parameters
| Field | Required | Description | Example |
|---|---|---|---|
| Name | ✅ | Identifier for the Source. Used when setting up Validators. | |
| Bucket | ✅ | Name of the S3 bucket to read data from. | |
| Prefix | ✅ | Prefix of the folder in the S3 bucket. | If “/a/b” is specified as a prefix, Validio only reads the files that are located in the “/a/b” folder |
| File pattern | Filter what files to read, based on file names and regex expressions. | file_chunk_number_\d+.csv | |
| Cron preset | Determines how often to query the bucket for new data based on a preset option. Select custom to use your own cron expression. | ||
| Cron expression | ✅ | Determines how often to query the bucket for new data based on cron expression. Expression of cron presets are displayed here. Used to enter your own cron expression. | |
| File format | ✅ | Select the type of file: CSV, Parquet, or JSON. For CSV file formats, specify the delimiter used in the CSV file and (optional) the character or string used to represent a null value. | CSV delimiter: , Null marker: NULL |
Updated 5 months ago