ChaosSearch provides the ability for live indexing with the integration for Google Pub/Sub. With the Pub/Sub Integration configured, ChaosSearch will be able to identify new files that have been put into the queue for discovery and indexing.
With Live Indexing enabled for an Object Group, ChaosSearch will monitor your GCP storage bucket for object creation events via user-specified PubSub subscription and index the newly-created objects, making the data available for querying as it becomes available.
GCP Service account
Please note that the GCP Service account configured for the ChaosSearch account must also be configured with appropriate access to the Pub/Sub subscription.
- In the GCP Management Console - Navigate to PubSub -> Create Topic
- Assign the topic a name - do not create a default subscription
- Click “Subscriptions” on the left menu, and create a subscription with the same name as your topic - increasing the Acknowledgement deadline to 600 seconds:
Alternately, you can do the above with the following command using Google’s gcloud utility:
gcloud --project YOUR-PROJECT-ID pubsub topics create YOUR-TOPIC-NAME gcloud --project YOUR-PROJECT-ID pubsub subscriptions create YOUR-SUBSCRIPTION-NAME --topic=YOUR-TOPIC-NAME --ack-deadline 600
- Create an object group
- Enter your Project ID and PubSub Subscription ID into the appropriate fields
- Click the Realtime box
- Click Create
- Click Start Indexing
- Navigate back to the Object Group you will now see Live Indexing configured
Connecting a storage bucket to the Pub/Sub queue must be done using Google’s gsutil command:
gsutil notification create -f json -p YOUR-LOGS-PREFIX -e OBJECT_FINALIZE -t YOUR-TOPIC-NAME gs://YOUR-BUCKET-NAME
Updated about 1 month ago