Object Group Isolation Keys

When you define an object group, you can configure isolation keys to define rules that will separate the index data into related "slices" based on rules that you define.

Isolation keys are an optional configuration setting, and are often used with live object groups. Each live object group is assigned a minimal reserve of dedicated worker resources to watch for and index new files. If you create three live object groups, three of those minimal reserves are allocated. When one live object group with an isolation rule can do the work to create the index data of multiple separate live object groups, it is a much more efficient use of system resources, and more workers remain available for other tasks like queries.

The following image shows a basic representation of different types of files saved in a customer's raw cloud-storage location. With isolation keys, one object group can index and separate the indexed data for its objects and create named isolation slices. These isolation key slices that can then be used as filters for Refinery views, to show only one, several, or all of the indexed data for that group.

When ChaosSearch indexes cloud-storage files that match an object group's filter rules, the resulting data is stored in separate slices that contain only the data related to an associated isolation key. Object group isolation is a means to separate ingestion into streams where indexing resources can be efficiently utilized.

Isolation keys can help administrators to manage visibility to data, and to be more efficient with their compute resources to support their live object groups.

How Isolation Keys Work

In the same way that object groups use regular expressions to filter the storage objects and find the objects to index, ChaosSearch uses a regular expression to define the isolation key rules for separating the filtered storage files into key-specific index files. For example, if your cloud storage has web site authentication log files stored in S3 using the following pathname format:

app/backup/<site>/auth-records-<date>.log.gz

You could create isolation keys to separate the indexed data using a regular expression to isolate by each unique site value, for example:

app/backup/(\S+?)/auth-records*.log.gz

Storage patterns could be used to isolate on values like customer IDs, business units, countries, corporate sites, platform regions, or applications. The regular expression rules can detect and start isolating for new values in the patterns. For example, if an expression is isolating by an application name used in a pathname, and a new application's logs are added to cloud storage and matches the regular expression, ChaosSearch automatically creates a new isolation key to the index the data related to that new application.

The following topics provide an overview of how to identify when object group configurations can benefit from isolation keys, and how to configure object groups to use isolation keys.

After you create object groups with isolation key rules, you can create Refinery views that show the data for only one or more isolation key slices, as described in Creating a View for Isolated Data. End users of that view can query or visualize the data for the associated isolation key(s), but not other data.