You can create many object groups to index various portions of the log and event files for your business data, though sometimes one object group can efficiently do the work of many. When does isolation make sense for your environment? Here are some factors to consider:

Are log and event files being sent to one cloud storage bucket?

• No -- Similar files are sent to different buckets for business reasons. (This indicates at least one object group per bucket.)
• Yes -- The same types of files are written to the same cloud storage bucket.

Do related groups of files within the cloud storage bucket have the same format (e.g., CSV, LOG, or JSON) and compression (e.g., GZIP, SNAPPY, NONE)?

• No -- Related files often have different formats and compressions. (This indicates at least one object group per format/compression option set per bucket.)
• Yes -- Related files in a storage bucket have the same format and compression.

Are files well-organized using repeatable pathname structures among the different groups/areas?

• No -- Related files often have different folder organizations within the bucket. (This indicates at least one object group per pathname style to find the matching files, per bucket.)
• Yes -- Related files in a storage bucket use a common pathname and folder structure.

If the answer to any of the factors above is a hard No (that is, the condition is unlikely or difficult to change), isolation might not be practical for your environment. Be sure to consult with ChaosSearch Customer Success because they have experience working with many different cloud storage organization styles. There are often some ways in which isolation can be used for efficiency and data separation. It may also be possible to adapt storage organization going forward to better support isolation and compute savings benefits.

Isolation Key Planning

Isolation key rules require that object storage files use deterministic, repeatable path names for the storage objects associated with an object group, and a predictable pattern to derive the isolation keys. When the path names and isolation keys are consistent, isolation key-based object groups work very efficiently.

When you configure isolation keys, you can use regular expression named capture group syntax to name the isolation key field with a user-familiar name. By default, ChaosSearch creates a cs_partition_key_0 field in the indexed data structure. The isolation key field is used in Refinery views to select the isolation slice(s) to use for visualization or querying, so a more familiar field name might be helpful for user who create views that use the isolation keys to filter the results.

For example, the following regex creates a named capture group Service as the isolation key column name that will contain the values for the different Web services in your log file paths. When users create a view for the object group, the Service column can be used as an isolation filter to limit results to one or more specific service name values in queries or visualizations that use that view:

logs/123456789012/WebServer/(?<Service>\\S+?)/.*

📘

Isolation capture group names must be unique.

If you plan to specify a regex capture group name to use instead of the default cs_partition_key_0 field, make sure that the capture group name is unique within the entire schema of the object group. The capture group name cannot conflict with a field that already exists in the object group.

Too Many Isolation Keys

Another important consideration is whether an isolation key pattern creates a reasonable number of slices, or a virtually unbounded number of slices.

If the isolation key regular expression causes ChaosSearch to create thousands or millions of separate keys/slices, isolation could incur a negative impact on query performance because of the very large number of segments to process. As a best practice, it is recommended to use rules that create less than 10,000 isolation keys for one object group.