📘
Filters validate individual datapoints - learn more here

Filter Overview

Datapoint Filters
Filter	Description
Range	Ensuring that a numeric feature is within or outside some interval.
Smart	Setting an adaptive numeric auto-threshold on a feature that adapts over time.
Threshold	Ensuring that a numeric feature is below or above some threshold
Relative time	Ensuring that two timestamp features in same the record have the desired relationship, e.g. one date is before the other
Boolean	Asserting the truthfulness of a boolean feature.
Duplicates	Ensuring that a feature does not include duplicates.
Enum	Ensuring that a string feature only contains values from a fixed set of specified values.
Null	Ensuring that a feature has a reasonable number of nulls

Set-up a filter

In order to set up a filter, navigate to the datapoint pipeline that you want to add the filter to. If you haven’t set up a datapoint pipeline yet, do so by following the steps found here. Once you’ve navigated to datapoint pipeline details, follow the steps below.

1. Start the Filter setup wizard

Click on the “New filter”-button.

2. Choose Filter

The wizard will guide you through the setup which consists of two steps. First step is to select which type of filter you want to set up. Refer to the table below to get an overview of the Filters and the pages to the left for details on the Filters.

3. Configure Filter

In the second step you will configure the filter, defining the pass/fail condition for the data points flowing through the filter. You also need to pick what target feature you want to use for this filter and a metric to use when configuring alerts (e.g. failing or passing percentage). The exact configuration parameters will depend on which filter you choose.

📘
In all filters you configure 'the allowed region'. E.g. if you configure a Range filter 'is between 1-10', values outside 1-10 will be filtered as anomalies

Include and exclude specific feature values

In addition to configuring the filter parameters you can also add inclusion/exclusion criteria for which partitions of the data the filter should be evaluated on. E.g. if there is a country column and a specific country should be included or excluded. This is done by selecting one or more of the features used for partitioning the data, and corresponding values for those features.

If only specific feature values are included, all other unspecified feature values will not be processed by the filter. Conversely, if only specific feature values are excluded, all other unspecified feature values will be processed by the filter.

Note that both include and exclude parameters can be set at the same time. In this case, include parameters take precedence over exclude parameters. Example: include records where the ‘country’ feature value is ‘Sweden’, out of the Sweden records, exclude records where ‘occupation’ is ‘Software Engineer’

After completing the filter setup wizard you can navigate to the filter’s details page to see a history graph of the computed metric and also continue setting up alerts to be triggered based on the computed metric. You navigate to the filter’s details page by clicking on the filter in the list (1).