Review the steps in this topic to create an object group that can use partition keys to segregate data into different partitions. Make sure that you have a pathname/regular expression to identify the target cloud storage files for the object group, and make sure that you have the regular expression that will define the keys used to separate the index data into the different partitions for querying.
To create an object group that uses partitioning:
- Follow the standard instructions to create an object group. Go to Storage and then click Create Object Group.
- In the Prefix and/or RegEx Filter field, type the values to filter the cloud object files that you want this object group to index.
In this example, the cloud object storage contains application log files that are generated for different regions in the cloud service. The files have pathnames in the format
<region> is the cloud service region, such as
-3. (This example will create a partition key based on the platform region folder names.)
- Click Object Filter to display the filter options.
- In the Partition By field, type the regular expression that you want to use for the partition value, or click the pencil icon to display a regular expression editor window in which you can construct and test a regular expression. A sample expression editor window follows.
- In the Regex field, review or type the regular expression to use for partitioning. This example uses the expression
cloudtrail/AWSLogs/250/CloudTrail/(\S.+)/20.*/*to partition log files by the region field.
Based on the object group filter, which included only the files that matched a pathname to files named
ap-northeast*, the partitions will be
-3 for this example).
In the Capture Groups area below the preview, look for a
Match statement with one of the partitioning values (usually the first found partitioning value) derived from the regular expression. If the Capture Groups area is empty, update the partitioning regex to make sure that it is correct.
- Complete the usual steps to create an object group by specifying its name, live indexing information, and options.
As shown in the sample window, the object group file filter and the partition key results in three partitions called
ap-northeast-3 based on the path names found in the cloud object storage. A sample object group Partitions tab follows:
The live object group watches for any new
ap-northeast-* files added to cloud storage and indexes those files as they are made available. If a new region like
ap-northeast-4 is added over time, a new partition is created for that region.
Updated 4 months ago
After you create a partitioned object group, follow these instructions to create a view that uses it: