Live Index Object Groups

When you create an object group, the Live indexing option configures ChaosSearch to watch for notifications of new files in cloud storage to index. Live indexing is the typical option for most environments that ship new log and event files to cloud storage buckets on a regular cadence.

Live indexing requires the configuration of an Amazon Web Services (AWS) SQS integration or Google Pub/Sub integration (depending on the cloud storage service used) to notify the indexing service that new objects are available. For more information, see Live Indexing - Amazon SQS or Live Indexing - Google PubSub.

To configure Live Indexing, select the Live Indexing option and paste in the ARN created for the AWS SQS queue, or the GCP Pub/Sub Project ID. A sample window follows:

If you do not select Live Indexing, the indexing style defaults to static indexing.

Static indexing configures ChaosSearch to run an indexing pass on existing cloud storage files in the specified bucket that match the object group rules. Static indexing starts when you click Start Indexing. This method is helpful when indexing files written previously to cloud storage (and thus no notifications would be sent for them). If new objects will be added to the bucket, you could use a live object group to index the new objects, or you could run a follow-up static indexing job to catch the new files.