Filters
Get an overview of all Filters available in Validio and learn how to set one up
Filters validate individual datapoints - learn more here
Filter Overview
Datapoint Filters | |
Filter | Description |
Range | Ensuring that a numeric feature is within or outside some interval. |
Smart | Setting an adaptive numeric auto-threshold on a feature that adapts over time. |
Threshold | Ensuring that a numeric feature is below or above some threshold |
Relative time | Ensuring that two timestamp features in same the record have the desired relationship, e.g. one date is before the other |
Boolean | Asserting the truthfulness of a boolean feature. |
Duplicates | Ensuring that a feature does not include duplicates. |
Enum | Ensuring that a string feature only contains values from a fixed set of specified values. |
Null | Ensuring that a feature has a reasonable number of nulls |
Set-up a filter
In order to set up a filter, navigate to the datapoint pipeline that you want to add the filter to. If you haven’t set up a datapoint pipeline yet, do so by following the steps found here. Once you’ve navigated to datapoint pipeline details, follow the steps below.
1. Start the Filter setup wizard
Click on the “New filter”-button.
2. Choose Filter
The wizard will guide you through the setup which consists of two steps. First step is to select which type of filter you want to set up. Refer to the table below to get an overview of the Filters and the pages to the left for details on the Filters.
3. Configure Filter
In the second step you will configure the filter, defining the pass/fail condition for the data points flowing through the filter. You also need to pick what target feature you want to use for this filter and a metric to use when configuring alerts (e.g. failing or passing percentage). The exact configuration parameters will depend on which filter you choose.
In all filters you configure 'the allowed region'. E.g. if you configure a Range filter 'is between 1-10', values outside 1-10 will be filtered as anomalies
Include and exclude specific feature values
In addition to configuring the filter parameters you can also add inclusion/exclusion criteria for which partitions of the data the filter should be evaluated on. E.g. if there is a country column and a specific country should be included or excluded. This is done by selecting one or more of the features used for partitioning the data, and corresponding values for those features.
If only specific feature values are included, all other unspecified feature values will not be processed by the filter. Conversely, if only specific feature values are excluded, all other unspecified feature values will be processed by the filter.
Note that both include and exclude parameters can be set at the same time. In this case, include parameters take precedence over exclude parameters. Example: include records where the ‘country’ feature value is ‘Sweden’, out of the Sweden records, exclude records where ‘occupation’ is ‘Software Engineer’
After completing the filter setup wizard you can navigate to the filter’s details page to see a history graph of the computed metric and also continue setting up alerts to be triggered based on the computed metric. You navigate to the filter’s details page by clicking on the filter in the list (1).
Updated about 2 months ago