A monitor will not run unless there is at least one trigger defined to that monitor. A monitor can have multiple triggers to check for, and take different actions, when various conditions are detected. For example, you could define one trigger to send a message to a destination like a Slack channel when a specific error message is detected once in the last 15 minutes of indexed data for a view. You might have another trigger that will page an administrator if that message is detected more than 5 times over the sixty minutes of indexed data.

Trigger conditions are painless scripts that return true or false to indicate if a trigger should be activated to send an alert. These scripts are given the extraction query response as JSON handed down in the ctx variable. Note that ChaosSearch supports a variety of common painless expression patterns for the typical trigger conditions and checks, but not all of the painless syntax is supported.

To define a trigger:

1. When you are creating or editing a monitor, scroll to the bottom of the window to the Triggers section.

2. Click Add trigger to open a panel with the fields for a new trigger.

3. In the Trigger name field, type a name for the trigger.
4. In the Severity level field, specify an alert severity level from 1 (Highest) to 5 (Lowest).
5. In the Trigger condition field, specify the conditions that must be met for the trigger to fire. The default condition is that the monitor query must return at least one hit/result. You can define a custom condition using painless style scripting. Click Info for more information about the scripting variables, and also see Trigger Condition Patterns below for some supported patterns.
6. In the Actions panel, define one or more actions to take when a trigger condition is met.

👍

An action is typical, but not required.

For example, you might skip an action to stage an alert, or to monitor conditions passively from the Alerting page, rather than via active notifications. It can be helpful for testing monitors, especially when a destination is not available for use at the time.

7. In the Action name field, type a name for the action that you are adding.
8. In the Destinations list, select an alert destination to which the alert will be sent.
9. In the Message subject field, type a clear message that will be sent in the alert.
10. In the Message field, update the template content as needed to provide helpful information for the alert consumer about the monitoring condition and the trigger detected.
11. Click Preview message to display a sample event message based on the information that you just specified. Note that the preview displayed in the web browser could diverge from the message as rendered by the service. It is helpful to use the Send test message feature to see a sample message processed by the service to confirm that the message renders as intended.
12. Under Throttling, you can select Enable action throttling if you want to limit the number of notifications you receive within a given time frame. Another field appears where you can specify the throttle timeframe of 1 to 1440 minutes.
    If a monitor checks a trigger condition every minute, you could receive one notification per minute. If you set action throttling to 60 minutes, you receive no more than one notification per hour, even if the trigger condition is met dozens of times during that hour.
13. You can choose to add another action if desired, for up to 10 actions on the monitor. For example, you might have one trigger to send a low priority alert if data crosses an early-warning threshold, and another trigger for a higher priority alert when data crosses a more severe threshold level.
14. When you finish specifying the trigger(s) for the monitor, save the monitor by clicking Create (or Update) at the bottom of the window.

Trigger Condition Patterns

The following patterns are some examples of typical evaluations that are supported in trigger conditions. They can be combined to form condition expressions in the painless script.

📘

Use care with complex condition expressions.

If you combine multiple patterns into evaluation expressions, note that there is no enforced limit on the number of terms that can be used in an expression. However, performance degradation has been observed when 6 or more patterns are contained in an expression. Note that ChaosSearch support is limited to single-line scripts, so an expression must be composed on one line.

Iterating over Result Collections

ChaosSearch includes a custom painless language extension called forall that supports the ability to iterate (or loop) over collections. The extension uses the following syntax style:

forall (i in ctx.results[0].aggregations.datehistogramagg.buckets) { i.doc_count > 0 }

In this example, if there is a collection of buckets from an aggregation such as ctx.results[0].aggregations.datehistogramagg.buckets, each bucket has a doc_count value, and you evaluate them using an expression—in this example, whether they all exceed a value of 0. The body of the expression can include the common painless comparison types that are supported.

❗️

Other painless looping methods are not supported.

The ChaosSearch implementation is a custom extension for iterating over collections. Painless includes some other looping mechanisms which are not expressions, and thus are not supported for use in ChaosSearch monitoring scripts.