Important Partitioning Considerations

Partitions can be helpful, but be sure to review these considerations.

Partition key rules require that object storage files use deterministic, repeatable path names for the storage objects associated with an object group, and a deterministic pattern to derive the partition keys. When the path names and partition keys are consistent, partitioned object groups work very efficiently.

When you configure partition keys for an object group, ChaosSearch adds the cs_partition_key_0 field to the indexed data structure. This is the field that will be used in Refinery views to select the partition(s) to use for visualization or querying.

Another important consideration is whether a partition key pattern creates a reasonable number of partitions, or a virtually unbounded number of partitions. If the partition key causes ChaosSearch to create thousands or millions of partitions, and thus thousands or millions of daily index files to scan, partitioning could incur a negative impact on query performance because of the very large number of index files to scan and process. As a best practice, it is recommended to use rules that create less than 10,000 partitions for one object group.

What’s Next

How to create partitioned object groups:

Did this page help you?