Configuring Object Group Partition Keys

How to create object groups that can segregate index data into partitions

Review the steps in this topic to create an object group that can use partition keys to segregate data into different partitions. Make sure that you have a pathname/regular expression to identify the target cloud storage files for the object group, and make sure that you have the regular expression that will define the keys used to separate the index data into the different partitions for querying.

Create an Object Group That Uses Partitioning

To create an object group that uses partitioning:

  1. Follow the standard instructions to create an object group. Go to Storage and then click Create Object Group.
  2. In the Prefix and/or RegEx Filter field, type the values to filter the cloud object files that you want this object group to index.
24962496

In this example, the cloud object storage contains application log files that are generated for different regions in the cloud service. The files have pathnames in the format cloudtrail/AWSLogs/id/<region>/<date>/<filename>.json.gz where <region> is the cloud service region, such as ap-northeast-1, -2, and -3. (This example will create a partition key based on the platform region folder names.)

  1. Click Object Filter to display the filter options.
21622162
  1. In the Partition By field, type the regular expression that you want to use for the partition value, or click the pencil icon to display a regular expression editor window in which you can construct and test a regular expression. A sample expression editor window follows.
11041104
  1. In the Regex field, review or type the regular expression to use for partitioning. This example uses the expression cloudtrail/AWSLogs/250/CloudTrail/(\S.+)/20.*/* to partition log files by the region field.

Based on the object group filter, which included only the files that matched a pathname to files named ap-northeast*, the partitions will be ap-northeast-1, -2, and -3 for this example).

20622062

In the Capture Groups area below the preview, look for a Match statement with one of the partitioning values (usually the first found partitioning value) derived from the regular expression. If the Capture Groups area is empty, update the partitioning regex to make sure that it is correct.

  1. Complete the usual steps to create an object group by specifying its name, live indexing information, and options.

As shown in the sample window, the object group file filter and the partition key results in three partitions called ap-northeast-1, ap-northeast-2, and ap-northeast-3 based on the path names found in the cloud object storage. A sample object group Partitions tab follows:

16761676

The live object group watches for any new ap-northeast-* files added to cloud storage and indexes those files as they are made available. If a new region like ap-northeast-4 is added over time, a new partition is created for that region.


What’s Next

After you create a partitioned object group, follow these instructions to create a view that uses it:

Did this page help you?