Advanced Detection configurations¶

Multiple detection rules¶

Multiple detection rules can be easily set up on the same metric.

For example:

In the following example, multiple detection rules is set up on the page_view metric. First, there is a percentage change rule. If the week-over-week change is more than 10 percent, detect it as an anomaly. Second, a threshold rule is set up, if the metric is less than 35800000, detect it as an anomaly. Third, algorithm detector is set up to detect anomalies automatically, also out of the anomalies detected by the algorithms, an anomaly filter is set up to say it’s an anomaly only if it compared to median over 4 weeks value, the change is more than 1%.

detectionName: test_yaml
description: If this alert fires then it means so-and-so and check so-and-so for irregularities
metric: page_view
dataset: business_intraday_metrics_dim_rt_hourly_additive

rules:
- detection:                      # Eg. Detect anomalies if the week over week change of this metric is more than 10%
    - name: detection_rule_1      # Give a unique name for this detection rule.
      type: PERCENTAGE_RULE       # Configure the detection type here. See doc for more details.
      params:                     # The parameters for this rule. Different rules have different params.
        offset: wo1w              # Compare current value with last week. (Values supported - wo1w, wo2w, median3w etc)
        percentageChange: 0.10     # The threshold above which you want to be alerted.
        pattern: UP_OR_DOWN       # Alert when value goes up or down by the configured threshold. (Values supported - UP, DOWN, UP_OR_DOWN)
- detection:
    - name: detection_rule_2
      type: THRESHOLD
      params:
        min: 35800000
  - name: filter_rule_2
    type: PERCENTAGE_CHANGE_FILTER
    params:
      offset: median4w
      threshold: 0.01

Anomaly filter rules¶

Each detection rule can be followed by multiple filter rules. The anomalies generated by detection rule must pass all the filter rules attached to it to be saved & sent.

For example:

- detection:
    - name: detection_rule_1           # Unique name for this detection rule
      type: THRESHOLD                  # The type for this detection rule or filter rule.
      params:                          # The parameters for this rule. Different rules have different params.
        min: 10000
  filter:
    - name: filter_rule_1
      type: PERCENTAGE_CHANGE_FILTER
      params:
        offset: wo1w
        threshold: 0.01
    - name: filter_rule_2
      type: PERCENTAGE_CHANGE_FILTER
      params:
        offset: median4w
        threshold: 0.01

This yaml will generate anomalies when the metric value is above 10000 and when compared to the week over week value, the change is above 1% and when compared to the median over 4 weeks value, the change is above 1%.

Anomaly filters are powerful tools to customize the anomaly detection.

To see the complete list of supported detection filter rules, click here :ref: `all-filter-rules.

Detection cron schedule¶

The cron schedule defines how often the detection runs and at what time it runs.

If the cron is not specified in the configuration, then ThirdEye will pick a default schedule. The default schedule depends on the data granularity. For daily metric, the cron is “0 0 14 * * ? *” which runs the detection at 14:00:00 pm UTC every day. For hourly metric, the detection is run at the beginning of every hour. For minute-level metric, the detection is run at every 15 minutes. (i.e. at minute :00, :15, :30, :45 of every hour.)

This can be configured to fit the data point generation time to reduce the time to detect the anomaly. For example, ” 0 0 0 * * ? * ” means do detection at 00:00:00 am UTC every day.

In the YAML this is configured at the level of a pipeline. Here is a good quartz cron schedule generator: https://www.freeformatter.com/cron-expression-generator-quartz.html

parameter name	supported values
cron	valid cron schedule

YAML format:

cron: 0 0 14 * * ? *   # Run daily at 2 PM UTC (7 AM PDT)

Deactivate the detection¶

Turn on/off the detection. If this parameter is missing, the detection will be turned on by default.

parameter name	supported values
active	true/false

YAML format:

active: false

Dimension exploration¶

ThirdEye supports creating alerts for each dimension value in a dimension. For example, you can set an alerts for all the values in countries, etc.

YAML format:

dimensionExploration:
    dimensions:
    - dimension_name_1 # name of the dimension

Explore dimension combinations is also supported. For example, if an alert is set for all combinations of platforms and countries, that means for each platform-country combination, there will be an detection set up. i.e. detection for ios-us, ios-uk, ios-fr, andorid-us, andorid-uk, android-fr, etc.

To help for finding the right dimensions, auto-complete is turned on for dimension values in ThirdEye YAML editor.

YAML format:

dimensionExploration:
    dimensions:            # a list of dimensions
    - dimension_name_1
    - dimension_name_2

Dimension filter¶

ThirdEye supports creating alerts for a filtered metric. For example, monitor the page views only in US. Multiple dimension filters is also supported.

The dimension name and dimension values have to be string. If the dimension name or value is double, interger, boolean value, etc, it need to be wrapped by double quotes.

YAML format:

filters:
    dimension_name:        # a list of dimensions
    - dimension_value_1
    - dimension_value_2

For example:

# monitor this metric where ip\_country\_code=us and browser in safari,
chrome, firefox.

# monitor this metric where ip_country_code=us and browser in safari, chrome, firefox.
filters:
    ip_country_code:
        - us
    browser:
        - safari
        - chrome
        - firefox

Data filter¶

ThirdEye supports filter on input data before running detection algorithm. E.g, You may not want to detect changes from 0.01 to 0.02, which is noisy although it is 100% change. It’s set under “dimensionExploration” section.

You don’t need to have dimension in order to set up data filter. E.g, you can set up data filter on inGraph data which doesn’t have dimension, but you need to set it under “dimensionExploration” section.

parameter name	description	supported values
minContribution	only monitor the dimension combinations contributes to overall metric is larger than the contribution	double value between 0 to 1
k	only monitor the dimension combinations contributes to overall metric is in top k.	integer
minValue	the aggregate value of this dimension combination must be larger than the threshold. If ‘dimensions’ field not set, this will apply to the overall metric.	double
minValueHourly	the aggregate value of this dimension combination hourly must be larger than the threshold. If ‘dimensions’ field not set, this will apply to the overall metric.	double
minValueDaily	the aggregate value of this dimension combination daily must be larger than the threshold. If ‘dimensions’ field not set, this will apply to the overall metric.	double
dimensionFilterMetric	The metric for dimension explore, dimension filter and data filter. If this value is not set, the metric used in dimension filter will be the same metric as the main metric. This can be different from a metric that the detection runs on(aka the main metric), but the dimension filter metric have to be in the same data set of the main metric.	String. Metric name in the same dataset as the main metric.

YAML format:

dimensionExploration:
    dimensions: # optional, only needed when your metric has dimension
    # a list of dimensions
    - dimension_name_1
    - dimension_name_2
    minContribution: 0.05 # only monitor the dimension combinations contributes to overall metric is larger than 5%
    k: 10 # only monitor the dimension combinations contributes to overall metric is in top 10
    minValue: 10
    minValueHourly: 20
    minValueDaily: 100
    dimensionFilterMetric: cold_signup # The metric for dimension explore, dimension filter and data filter. Can be a different metric.

Anomaly Merging¶

ThirdEye will merge anomalies for the same metric & dimension if they are overlapped with each other. The merger’s behavior is configurable from yaml.

parameter name

description

supported values

default value

maxGap

The gap in milliseconds between two anomalies in order to be merged. If the gap between two anomalies is less than this value, they will be merged into one anomaly.

long

7200000

(2 Hours)

maxDuration

the maximum allowed duration of a merged anomaly.

long (Must be >= 900000 (15 mins))

MAX_VALUE

For example:

merger:
  maxGap: 3600000 # set the gap of merging to be 1 hour.
  maxDuration: 86400000 # set the longest anomaly allowed to be one day

Advanced Subscription Group configurations¶

Enable/Disable Notification¶

Disabling an alert notification will disable all and any kind of alerting scheme that you may have configured. You will completely stop receiving notifications.

# Enable or disable notification of alert

active: true

Note: As long as the detection is not disabled, we will continue to detect anomalies and display them on the ThirdEye UI but you will not be notified.

Modify Alert Email Subject¶

Different teams use Third Eye for monitoring different scenarios. For some who monitor a dataset, they prefer to have the dataset name in the alert email title, others who monitor metrics would like to have the metric or the subscription group name in the alert title. By default, we include the metric name in the alert email title. If you wish to configure it differently, then include the below line in your subscription config.

emailSubjectStyle: ALERTS # Allowed values - [ALERTS, METRICS, DATASETS]

Parameter	Description	Alert Email Subject Template	Example: Alert Email Subject
ALERTS	Only uses the subscription group name in the alert email subject	Thirdeye Alert : my_subscription_group_name	Thirdeye Alert : m2g_alert_monitoring
METRICS	Includes the name of the metrics which had an anomaly in the email subject.	Thirdeye Alert : my_subscription_group_name - <comma-separated-list-of-metrics>	Thirdeye Alert : m2g_alert_monitoring - cold_signup
DATASETS	Includes the dataset names of the metrics which had an anomaly in the alert email subject.	Thirdeye Alert : my_subscription_group_name - <comma-separated-list-of-datasets>	Thirdeye Alert : m2g_alert_monitoring - business_intraday_metrics_dim_rt_hourly_additive

Enable/Disable Alert Suppression¶

Alerts can be suppressed to avoid generating too many alert notifications especially during maintenance windows, deployments, or unexpected variations in metric during holidays like Thanksgiving, Christmas, or during the beginning/end of quarter when metric usually spikes, etc.

Completely suppress alert for a period¶

As the title says, if you use the below configs in the subscription config, you will not receive any alerts for the configured duration.

alertSuppressors:
- type: TIME_WINDOW                     # Suppresses all anomalies/alerts which START after windowStartTime and before windowEndTime
  params:
    windowStartTime: 1542888000000      # The suppression window start time in epoch millis.
    windowEndTime: 1543215600000        # The suppression window end time in epoch millis.

Partially suppress alert for a period after applying some thresholds¶

If you do not want to completely suppress an alert and also not prefer receiving a lot of alerts especially during long shutdown periods like Christmas, then you can opt to receive only the most severe alerts. To configure this, you would need to let Third Eye know a reasonable window based on some thresholds within which anomalies can be ignored.

alertSuppressors:
- type: TIME_WINDOW
  params:
    windowStartTime: 1542888000000
    windowEndTime: 1543215600000
    isThresholdApplied: true
    expectedChange: -0.25               # Expect the metric to drop by 25 percent during the configured time window
    acceptableDeviation: 0.10           # Any variation of 10 percent up/down after the drop is reasonable and anomalies detected within this window can be ignored.

Note: Suppression only suppresses the notification of the anomaly. Third Eye will continue to run anomaly detection even during the suppression window, detect anomalies and automatically label them as True Positive.