HomeRequest DemoContact
HomeRequest DemoContact

Object storage

Validio supports major Object storage that modern data teams work with today.

General considerations

Consider the following when you read and validate fields identified in CSV files from an object storage:

  • When you configure an Object storage source, you must specify a storage folder.
  • Validio reads files from the folder and it's nested folders recursively.
  • Validio reads files in order from earliest to the most recent time when they were modified.
  • You can use regex expressions to filter what files to read.
  • The CSV schema must be consistent across all files within the specified top level folder:
    1. For each of the files, the connector expects the same fields and field names in the same order.
    2. Missing fields are interpreted as empty fields, which have consequences on the analytics involving those fields.

Detect new files in Object storage

Validio detects new files based on the file timestamp attribute that suggests the last updated time. This attribute is specific to each Object storage provider:

The Validio schema also includes the validio_file_created_at field, which contains the timestamp generated when a file is created in the Object storage. The timestamp is in RFC3339 UTC "Zulu" format.

Cost and performance

Costs associated with reading data from Object storage:

  • If the traffic crosses cloud regions, there are potential network costs between Validio and the Object storage.
  • The costs for listing objects are negligible.