Partitioning Overview

An overview of partitioned object groups and their benefits

You create object groups to define the cloud object storage files to index by using object name prefixes and/or regular expressions to uniquely identify the files.

One object group can filter and index many source objects that have the same format and compression using the same index rules. You could create many separate object groups to index different collections of source files, but this is typically best when source file format schemas are different, the input source file filters are different, or if factors like compression or indexing rules are different.

13041304

If the format, compression, and indexing rules are the same, using many object groups to do the work that could be done by one group is a less efficient use of system resources. For live object groups, each with a small reserve of compute resources, that minimal set of resources can add up quickly, limiting the compute resources available for other tasks like querying.

When it is possible to use one live index group instead of multiple groups, the partition key feature can keep the index data segregated in partitions for data isolation and visibility/analysis needs.

13141314

In the same way that object groups use regular expressions to filter the storage objects to process, ChaosSearch uses a regular expression to define a partition key that can segregate the index data into discrete partitions. For example, if the web site authentication log files all have the pathname format:

app/backup/<site>/auth-records-<date>.log.gz

You could configure a partition key using a regular expression for the unique site folder value, for example:

app/backup/(\S.+)/auth-records*.log.gz

As other options, you could partition based on file name prefixes, like auth-records, error-records, write-records, and so forth, application names, or similar unique patterns within the object storage files.

Carefully plan the regular expressions for the object group filter and for the partition key rule to ensure that you have a good storage object file match for the overall group, and that the partition key expression creates the correct partitions that will be used by the Refinery views (created later) that users will access.


What’s Next

Choosing to partition, or not:

Did this page help you?