Object groups include support for using keys to separate index data into isolated streams. When ChaosSearch indexes customer files that match an object group's filter rules, the resulting data is stored in separate index partitions that contain only the data related to an associated key. Object group partitioning is a means to separate ingestion into isolated streams where indexing resources can be efficiently utilized.
Partition key rules are customer-specified regular expressions that derive the keys from patterns in the storage object file pathnames, like folder names or file name prefixes. These patterns identify entities like customer IDs, business units, countries, corporate sites, platform regions, or applications. The data for each entity might require separation for security reasons, or for analysis reasons to organize related data more easily for end users. The regular expression rules are also able to detect new patterns. If storage files are written with new key values (such as files with a new customer ID), ChaosSearch automatically creates a new file partition for the index data related to that new key.
Partition keys are an optional configuration setting, and are often used with live object groups. Each live object group is assigned a minimal reserve of dedicated worker resources to watch for and index new files. If you create three live object groups, three of those minimal reserves are allocated. When one live object group with a partition rule can create partitioned index data and eliminate the need for separate groups, this is a more efficient use of system resources, and more workers remain available for other tasks like queries as needed.
The following topics provide an overview of how to identify when object group configurations can benefit from partitions, and how to configure object groups to use partition keys.
After you create object groups with partition rules, you can create Refinery views that show the data for only one or more partitions, as described in Creating a View for Partitioned Data. End users of that view can query or visualize the data for the associated partition(s), but not other data.
Updated 4 days ago
Read more about the optional partitioned object groups: