Kinesis
Prerequisite credentials
You need the following to configure Kinesis credentials in the Validio platform:
- A IAM user with permissions to access the specified Kinesis stream
- Credentials for the IAM user
Permissions for IAM user
Certain permissions and roles are required for the IAM user.
For more information, refer to Actions, resources, and condition keys for Amazon Kinesis.
IAM user for Source config
We recommend that you create an IAM user with access to read records in a stream, for the Validio platform to ingest data from your Kinesis stream.
The following permissions must be assigned to your IAM user:
- Action
DescribeStreamSummary
; access levelRead
; resource typestream
- Action
ListShards
; access levelList
- Action
GetShardIterator
; access levelRead
; resource typestream
- Action
GetRecords
; access levelRead
; resource typestream
Access key
You must supply an Access key and secret key to authenticate to the S3 bucket.
For more information, refer to Managing access keys for IAM users.
Credential parameters
Field | Required | Description | Example |
---|---|---|---|
Name | β | Identifier for the credentials. Used when accessing sources. | East coast weather forecast |
Access key | β | Access key with permissions to access the Kinesis stream. | AKUEBNDJHA7BJSHJVA6F |
Secret key | β | Secret key for the specified access key. | aslk/KOJHJKBuibhj 2jq3csaVGHYTFgvhc ascuyg |
Ingestion in JSON format
Validio currently supports ingesting a Kinesis record in JSON format.
Configuration parameters
Field | Required | Description | Example |
---|---|---|---|
Name | β | Identifier for the connector. Used when setting up Validators. | |
AWS region | β | Region that the Kinesis stream is available in. | eu-north-1 |
Stream name | β | Name of the stream to ingest data from. | east-coast-weather |
Inferring a schema
Based on your data, Validio helps to infer a schema of your source connector.
The automatic schema inference requires events in the stream when you connect your source. The stream can be considered empty by the Validio platform in the following circumstances:
- A steam has no events when connecting to Validio. For example, all events are deleted according to the retention period, and no new events have been published.
Updated 9 days ago