Stream processing

General considerations

The following are general considerations regarding how the Validio platform ingests data from streams:

  • Message structure needs to be consistent across messages in a topic, i.e. consistent fields. Additional fields since the creation of the connector will be ignored. Missing fields will be interpreted as empty fields, but will have consequences on the analytics involving those fields

The fields identified in the streamed records will then be available to perform data quality validation on.