1. Find the ‘New dataset pipeline’ button on the Pipeline Page
- Streaming Datasets A batch defined by specified number of records/datapoints
- Data Warehouse Datasets: Table divided into batches by specified time period, e.g. records within a five second window pertains to the same batch
- Cron Datasets: Schedule batches based on cron expressions
- Object Store Datasets: Logical batches by file/BLOB, e.g. a CSV file is a batch
Learn more about the configuration parameters on their dedicated sub-pages. What all type of dataset pipelines have in common is that the user is able to select a reference source at this stage to enable reference monitors.
New to partitioning? Learn more about pipeline partitioning and why it is one our most used features
Updated 5 months ago