Define Triggers for a Monitor
Add one or more triggers to a monitor to enable the monitor and to take action when a defined condition occurs
A monitor will not run unless there is at least one trigger defined to that monitor. A monitor can have multiple triggers to check for, and take different actions, when various conditions are detected. For example, you could define one trigger to send a message to a destination like a Slack channel when a specific error message is detected once in the last 15 minutes of indexed data for a view. You might have another trigger that will page an administrator if that message is detected more than 5 times over the sixty minutes of indexed data.
Trigger conditions are painless
scripts that return true or false to indicate if a trigger should be activated to send an alert. These scripts are given the extraction query response as JSON handed down in the ctx
variable. Note that ChaosSearch supports a variety of common painless expression patterns for the typical trigger conditions and checks, but not all of the painless
syntax is supported.
To define a trigger:
- When you are creating or editing a monitor, scroll to the bottom of the window to the Triggers section.
- Click Add trigger to open a panel with the fields for a new trigger.
- In the Trigger name field, type a name for the trigger.
- In the Severity level field, specify an alert severity level from 1 (Highest) to 5 (Lowest).
- In the Trigger condition field, specify the conditions that must be met for the trigger to fire. The default condition is that the monitor query must return at least one hit/result. You can define a custom condition using
painless
style scripting. Click Info for more information about the scripting variables, and also see Trigger Condition Patterns below for some supported patterns. - In the Actions panel, define one or more actions to take when a trigger condition is met.
An action is typical, but not required.
For example, you might skip an action to stage an alert, or to monitor conditions passively from the Alerting page, rather than via active notifications. It can be helpful for testing monitors, especially when a destination is not available for use at the time.
- In the Action name field, type a name for the action that you are adding.
- In the Destinations list, select an alert destination to which the alert will be sent.
- In the Message subject field, type a clear message that will be sent in the alert.
- In the Message field, update the template content as needed to provide helpful information for the alert consumer about the monitoring condition and the trigger detected.
- Click Preview message to display a sample event message based on the information that you just specified. Note that the preview displayed in the web browser could diverge from the message as rendered by the service. It is helpful to use the Send test message feature to see a sample message processed by the service to confirm that the message renders as intended.
- Under Throttling, you can select Enable action throttling if you want to limit the number of notifications you receive within a given time frame. Another field appears where you can specify the throttle timeframe of 1 to 1440 minutes.
If a monitor checks a trigger condition every minute, you could receive one notification per minute. If you set action throttling to 60 minutes, you receive no more than one notification per hour, even if the trigger condition is met dozens of times during that hour. - You can choose to add another action if desired, for up to 10 actions on the monitor. For example, you might have one trigger to send a low priority alert if data crosses an early-warning threshold, and another trigger for a higher priority alert when data crosses a more severe threshold level.
- When you finish specifying the trigger(s) for the monitor, save the monitor by clicking Create (or Update) at the bottom of the window.
Trigger Condition Patterns
The following patterns are some examples of typical evaluations that are supported in trigger conditions. They can be combined to form condition expressions in the painless
script.
Use care with complex condition expressions.
If you combine multiple patterns into evaluation expressions, note that there is no enforced limit on the number of terms that can be used in an expression. However, performance degradation has been observed when 6 or more patterns are contained in an expression. Note that ChaosSearch support is limited to single-line scripts, so an expression must be composed on one line.
Pattern | Summary | Symbol |
---|---|---|
"string" | A raw string value | string |
9999.0 | A raw number | number |
\+,-,/,\* | An arithmetic operation | op |
> ,>=,==,\<,\<= | Arithmetic signs to compare two numbers | sign |
&&,|| | AND OR boolean values | and/or |
() | Precedence of expression evaluations | |
ctx.results[0].hits.total.value | How to access the total number of results | mt_group_doc_count |
ctx.results[0].hits.hits[0]._source.NUMKEY | NUMKEY is the name of a number field spelled out even if that field contains dots or [] square brackets. | rowaccess |
ctx.results[0].hits.hits[0]._source.STRKEY | STRKEY is the key to a string field. The hits array is accessed at the "zero" index. In painless, this matches to the first result in the array. | mt_group_agg |
ctx.results[0].aggregations.MTGROUPAGG.value | How to access an aggregation with an “empty grouping” that is an ElasticSearch aggregation and does not separate values into multiple buckets. Specifically, sum's, average's, min's, and max's. | mt_group_agg |
ctx.results[0].aggregations.GROUPING1.buckets[0] ...GROUPINGK.buckets[0].doc_count | How to access the doc_count of an ElasticSearch aggregation named GROUPING that creates buckets. Specifically (date) histogram and terms aggregations. | group_doc_count |
ctx.results[0].aggregations.GROUPING1.buckets[0] ...GROUPINGK.buckets[0].key | How to access the key of a grouping aggregation (with any level of nesting starting at GROUPING1 to GROUPINGK). | group_key |
ctx.results[0].aggregations.GROUPING1.buckets[0] ...GROUPINGK.buckets[0]. | How to access bucketed subaggregations where GROUPING1, GROUPING2 … GROUPINGK are the names of grouping aggregations and MTGROUPAGG is the name of the metric at the end. | group_agg |
return true | A trigger condition that’s always true so long as there are some results. | true |
return false | Always false | false |
mt_group_doc_count op number sign number number op mt_group_doc_count sign number | Arithmetic on the total number of results compared to a number. | mt_group_doc_count_cmp |
rowaccess op number sign number number op rowaccess sign number rowaccess sign number | Row result access to number fields compared to another number potentially with arithmetic. | rowaccess_cmp |
mt_group_agg group_doc_count op number group_key op number group_agg op number | Arithmetic on any aggregated value. | agg_arithmetic |
strrowaccess == string | String comparison to row results. | strrowaccess_cmp |
rowaccess op rowaccess sign number | Arithmetic with two different fields in row access. | rowaccess_arithmetic |
rowaccess_cmp,strrowaccess_cmp and/or rowaccess_cmp,strrowaccess | Combining two row access comparisons with an and/or. | |
group_doc_count sign number group_key op number mt_group_agg sign number group_agg sign number agg_arithmetic sign number | A comparison of an aggregated value to some number. | agg_cmp |
group_agg,group_doc_count op group_agg,group_doc_count sign number | Arithmetic with two different aggregated values. IMPORTANT These aggregations have to be at the SAME level of nesting. | same_agg_plan_arithmetic |
simple_agg_cmp and/or simple_agg_cmp | Combining two simple aggregation comparisons. IMPORTANT These aggregations have to be at the SAME level of nesting. | same_agg_plan_andor |
Iterating over Result Collections
ChaosSearch includes a custom painless
language extension called forall
that supports the ability to iterate (or loop) over collections. The extension uses the following syntax style:
forall (i in ctx.results[0].aggregations.datehistogramagg.buckets) { i.doc_count > 0 }
In this example, if there is a collection of buckets from an aggregation such as ctx.results[0].aggregations.datehistogramagg.buckets
, each bucket has a doc_count
value, and you evaluate them using an expression—in this example, whether they all exceed a value of 0
. The body of the expression can include the common painless
comparison types that are supported.
Other
painless
looping methods are not supported.The ChaosSearch implementation is a custom extension for iterating over collections. Painless includes some other looping mechanisms which are not expressions, and thus are not supported for use in ChaosSearch monitoring scripts.
Updated 30 days ago