Alerting Overview

ChaosSearch and its Search Analytics interface helps you to search across and visualize the data from your log and event files, to detect conditions such as a peak or drop in volume, an error message, or a behavior that service administrators would want to investigate.

Alerting is a Search Analytics feature that automates the detection of these types of important conditions, and can proactively send notifications to a team's Slack channel or to a monitoring tool using a custom webhook interface. When you apply monitors to views for Live Index groups, the system can raise an alert when conditions are detected in newly indexed data.

Use the Search Analytics > Alerts page to review a list of all alerts for detected conditions, and to manage the monitors and destinations that configure the monitoring rules. Users can also acknowledge an alert to inform other users that the situation is being reviewed.

The alerting process requires the following resources:

• Monitors define a condition or behavior to watch. You can define a monitor with an extraction query or a visual graph.

• Triggers specify thresholds or boundaries at which the condition becomes a concern, how frequently to run the check for the condition, and an optional destination for a notification. A monitor must have at least one trigger to be enabled. A monitor could have up to 10 triggers to define different levels and severities or destinations. When a monitor condition is detected, an alert enters the Active state. While active, monitors will continue to check for the condition on the scheduled interval, and send configured notifications while the condition is true.

• Destinations define a location to which a notification is sent when an alert is active. You can send messages to a Slack channel or to a designated application via a custom webhook. Alerts are always posted to the Search Analytics Alerts page.

Best Practices for Alerts and Notifications

It can be easy to under-configure alerts and miss out on important conditions, or to over-configure them and distract teams with too many alerts to investigate. As a good practice, be sure to plan for the alerts that are most helpful, and set the proper "radar" for teams to see and respond to them. As part of the alert planning, be sure to consider:

• The important conditions to monitor and how frequently to check for them based on the condition's impact and the ingest rate of the data that contains the condition information
• The correct destination, if any, to notify the appropriate personnel when conditions occur
• The message information and severity designation to help responders prioritize alerts and take action

A well-structured alerting plan can help teams to manage alerts proactively, use resources most efficiently, and improve end-user experience with faster detection and time-to-resolution for the most severe conditions.

👍

Avoid the Alert Firehose Effect

As a good practice, start with a smaller set of very specific alert conditions, then tune and grow the monitors over time as you identify the conditions that are most important and detectable from your log and event data.

For assistance with your alert configuration, contact your ChaosSearch Customer Success representative.