Object Group Isolation Keys

Isolation keys help administrators to manage visibility to data, and to be more efficient with their compute resources to support their live object groups.

Isolation keys are an optional configuration setting. When you define an object group to index data, you can define isolation rules that will separate the indexed data for matching raw source files into separate daily intervals as the data is written into storage.

Isolating the indexed data helps to keep information for related files together for analysis and separated from other, different, files and areas. The isolation keys uniquely identify each isolated interval of data and can be used as a filter to specify which subset(s) of data to include in views used for searches/queries or visualizations.

From a performance perspective, each live object group is assigned a minimal reserve of dedicated worker resources to watch for and index new files. If you create three live object groups, three of those minimal reserves are allocated. If one live object group with an isolation rule can do all of the work to index data instead of using multiple live groups with identical processing instructions, it is a much more efficient use of the object group reserve, and more workers remain available for other tasks like queries.

The following image shows a basic representation of different types of log files saved in a customer's raw cloud-storage location. With isolation keys, one object group can index and separate the indexed data for its objects and create named isolation slices.

The resulting indexed data is stored in separate slices that contain only the data related to an associated isolation key. Thus,

How Isolation Keys Work

In the same way that object groups use regular expressions to filter the storage objects and find the objects to index, ChaosSearch uses a regular expression to define the isolation key rules for separating the filtered storage files into key-specific index files. For example, if your cloud storage has web site authentication log files stored in S3 using the following pathname format:

app/backup/<site>/auth-records-<date>.log.gz

You could create isolation keys to separate the indexed data using a regular expression to isolate by each unique site value, for example:

app/backup/(\S+?)/auth-records*.log.gz

Storage patterns could be used to isolate on values like customer IDs, business units, countries, corporate sites, platforms, regions, or applications. The regular expression rules can detect and start isolating for new values in the cloud storage pathname patterns. For example, if an expression is isolating by an application name used in a pathname, and a new application's logs are added to cloud storage and matches the regular expression, ChaosSearch automatically creates a new isolation key to the index the data related to that new application.

The following topics provide an overview of how to identify when object group configurations can benefit from isolation keys, and how to configure object groups to use isolation keys.

After you create object groups with isolation key rules, you can create Refinery views that show the data for only one or more isolation key slices, as described in Creating a View for Isolated Data. End users of that view can query or visualize the data for the associated isolation key(s), but not other data.