Pre-requisites

You will need:

  • An IAM user with permissions to access the specified S3 bucket
  • Credentials for the IAM user. Either:
    • Access key and secret key
    • Temporary security credentials token

IAM user

It is recommended you create an IAM user with read-only access for the Validio platform to ingest data from your S3 bucket. Details of these permissions can be found on S3 here.

The following permissions are required:

  • s3.ListBucket
  • s3.GetObject

Credentials

You must supply either an access key and secret key or a temporary security credentials token.

  • Information on obtaining an access key and secret key can be found here
  • Information on how to obtain a temporary security credentials token can be found here

S3 configuration parameters

Field Required Description
Name Identifier for the connector. Used when setting up pipelines.
Region Region that the S3 bucket is available in.
Bucket Name of the S3 bucket to ingest data from.
Access key ✅* Required if not using temporary security credentials token. The access key used in combination with the secret key for the Validio platform to access the bucket with.
Secret Key ✅* Required if not using temporary security credentials token. The secret key used in combination with the access key for the Validio platform to access the bucket with.
Token ✅* Required if not using access/private key. The token for the Validio platform to access the bucket with.
Folder Name of the folder in the bucket to ingest data from.
Endpoint Endpoint of AWS service. It is useful for testing with Amazon Localstack
Polling interval value Determines how often to query the bucket for new data.
Unit The time unit used for the polling interval. E.g. a polling interval value of 2 with a unit of “hours” will poll every two hours.

[*] Depending on which method of credentials are used