A monitor is a job that runs on a defined schedule and queries for conditions or behaviors. The results are used as input for one or more triggers. You create monitors to specify the types of conditions that you want to watch for possible alerting and notifications to take action.

Creating a Monitor

To create a monitor:

1. In Analytics > Alerting, click the Monitors tab. The Monitors page opens.

2. Click Create monitor. The Create Monitor window opens.

3. Type a name for the monitor.
4. In the Monitor state field, you can select to disable the monitor if you are not yet ready to use it. The monitor must be enabled before it can take effect.
5. In the Method of definition field, select how to you want to define the monitor:

• Select Define using visual graph to create a monitor that watches for when a value is above or below a threshold for a period of time.
• Select Define using extraction query if you want more flexibility for the conditions that you want to query for (using the Elasticsearch query DSL) and how you evaluate the results of that query.

6. In the Select index field, select the ChaosSearch index view on which you are basing the monitor.
7. In Time field, which appears if you selected the visual graph option, select the column that you want to use for the time field.
8. In the Query performance field, review the query duration, and related information.
9. In the Monitor schedule section, select the frequency on which to run the query (such as By interval, Daily, Weekly, Monthly, or a Custom cron expression). Depending on the frequency option, the window displays more granular selection options.
10. Click Create to define the monitor.

🚧

Monitor Permissions and Users

When you create and save a monitor, the monitor definition is updated with the information for the ChaosSearch groups associated with your user account. Use caution when reviewing monitor definitions, because saving a monitor as a different user could break the monitor. If RBAC group assignments change, or if permissions assigned to the RBAC groups used for a monitor change, the monitor might not work after those updates.

See the troubleshooting section for more information.

Review the following sections for more details about the monitor options.

Defining an Extraction Query Monitor

If you select the option to Define using extraction query, as in the following example, the window updates to show new fields for the query.

Select the index view to monitor in the Index dropdown. The window updates to display two new fields. The left column Define extraction query allows you to define the extraction query, and the Extraction query response column on the right updates to show the relevant fields for the extraction query.

Click Run to populate the column on the right. The response in the right column produces the values from the selected index that could be added to the extraction that you are building in the left column.

📘

Tool Tip

When defining the extraction query, it is important to ensure the "match_all" query is correctly defined. Creation will not take effect unless it is properly set.

A sample match follows:

When the appropriate values have been added to the extraction query, click Create.

Define Using a Visual Graph

If you select the option to Define using visual graph, as in the following example, the window updates to show new fields for the query.

Select the Refinery view to monitor in the Index dropdown, then select the Time field to use for the query from one of the possible timestamp fields in the index view. The window updates to display a graph area, and populates the recent data for the graph.

If a visual graph does not appear, there may not be any values for the selected time period. The sample chart defaults to the last hour, but you might need to adjust that timeframe.

Troubleshooting Monitor Authorization Permissions

If a configured monitor that was working previously begins to raise an alert with the following message, there is a groups permission error to troubleshoot:

Error: chaossumo.util.akka.http.ChaosDirectives$Exceptions$AuthorizationException$: Authorization failed.

The problem could be that the groups for the user that created or last updated the monitor did not have the
kibana-opendistro:alerting:alerts:read permission. The monitor does not have permission to run. There could also be an issue where the RBAC groups associated with the user who last saved the monitor changed and no longer have the proper permissions to use the views, object groups, or query associated with the monitor. Or, the groups associated with the monitor definition changed when a user updated the monitor.

When troubleshooting this error, it can be very helpful to obtain the groups that are configured for the monitor that raised the alert, so that one or more groups could be investigated for the alerts permission. One way to obtain the group IDs for a monitor is to use the Browser DevTools window to display more information about the monitor.

1. Navigate to the Monitors page and open the DevTools window.
2. Select the monitor that triggered the alert. You should see a Name with the same Monitor ID value in the DevTools left frame of the Network tab.
3. Select the monitor ID name element, then click Preview. The right pane updates with information about the resource. Click to expand the groupIds property.

In the example above, groupIds is set to default, which is common for monitors created by the root user, especially during the ChaosSearch trial phase. During the production transition, the default group is usually updated to have a smaller set of basic permissions for new users who are not otherwise assigned to groups. Each site administrator typically creates new groups for the production environment to specify the various levels of user access that are needed. The resolution for this problem is to update and save the monitor while logged in as a user who has proper group assignments with the full complement of permissions so that the monitor can run successfully. As an alternative, if the current user is the person who must manage the monitors, the solution is to ensure that the monitor administrator has the correct group assignments to fully manage and run monitors.

Sometimes the groupIds assigned to a monitor are a sequence of one or more internal group IDs for the user who last saved the monitor. In this case, if a monitor sends Authorization alerts, a group is missing the alerts permission, or the user might not be assigned to the correct groups needed for the monitor to run. The group IDs list can provide the ChaosSearch Customer Success team member with information needed to diagnose the root cause of the authorization alert.