Object Store

General considerations

The following are general considerations regarding how the Validio platform ingests data from object stores:

  • As part of the configuration of an object store connector, the user needs to specify a folder. Validio will ingest files from this folder recursively, i.e. nested folders are also included
  • Validio currently supports CSV files
  • Files are read in lexicographic order
  • The CSV layout must be consistent across all files within the specified top level folder. The connector expects a constant number of fields across these files with the same field names. Additional fields since the creation of the connector will be ignored. Missing fields will be interpreted as empty fields, but will have consequences on the analytics involving those fields

The fields identified in the CSV files will then be available to perform data quality validation on.