Datapoint Pipeline Configuration

1. Find the ‘New datapoint pipeline’ button on the Pipeline Page

30803080

2. Configure the datapoint pipeline

30803080
Parameter name and description Parameter value
1. Name Arbitrary String
2. Source List of configured Source connectors + metadata connectors
3. Destination (optional)

Destination to sink out erroneous data.

List of configured Destination connectors
4. Notification rule (optional)

Note: Without a notification rule, alerts will be visible in the platform UI, but not sent as a notification to a notification channel, e.g. Slack

List of configured Notification rules
5. Timeout (seconds)

Parameter for sessionized datapoint batch.

If the time since the last ingested record exceeds the specified wait interval, a new datapoint batch is created

Integer value
6. Maximum session time (seconds)

Parameter for sessionized datapoint batch.

If Timeout is never triggered, maximum session time forces a new datapoint batch to be created after Maximum session time. Maximum session time starts counting from the oldest un-batched datapoint.

Integer value
5. Data time feature (optional)

Empty: Order of the records/datapoints will be determined by the time of ingestion of the data into Validio

Filled in: In case of backfilling (i.e. ingesting historical data), timestamps for the records are needed in order for Validio to correctly order the records

List of features with date formats in Source

3. Choose which feature(s) to partition on, if any

30803080

Simply check box the features you want to partition on

📘

New to partitioning? Learn more about Partitioning Pipelines and why it is one our most used features


Did this page help you?