Live Indexing - Google Pub/Sub

Google PubSub is a fully managed message publish/subscription service that enables you to decouple and scale microservices, distributed systems, and serverless applications.

What does ChaosSearch offer GCP PubSub users?

ChaosSearch provides the ability for live indexing with the integration for Google Pub/Sub. With the Pub/Sub Integration configured, ChaosSearch will be able to identify new files that have been put into the queue for discovery and indexing.

Functionality of Live Indexing

With Live Indexing enabled for an Object Group, ChaosSearch will monitor your GCP storage bucket for object creation events via user-specified PubSub subscription and index the newly-created objects, making the data available for querying as it becomes available.

Add PubSub Integration in ChaosSearch - Quick Option

🚧

GCP Service account

Please note that the GCP Service account configured for the ChaosSearch account must also be configured with appropriate access to the Pub/Sub subscription.

Add Pub/Sub Integration in ChaosSearch

  1. In the GCP Management Console - Navigate to PubSub -> Create Topic
  2. Assign the topic a name - do not create a default subscription
  1. Click “Subscriptions” on the left menu, and create a subscription with the same name as your topic - increasing the Acknowledgement deadline to 600 seconds:

Adding Pub/Sub Subscription to ChaosSearch Object Group

  1. Create an object group )
  2. Enter your account ID, and PubSub subscription name into the appropriate fields
  3. Click Create
  4. Click Start Indexing
  1. Navigate back to the Object Group you will now see Live Indexing configured

Alternately, you can do the above with the following command using Google’s gcloud utility:

gcloud --project YOUR-PROJECT-ID pubsub topics create YOUR-SUBSCRIPTION-NAME
gcloud --project YOUR-PROJECT-ID pubsub subscriptions create YOUR-SUBSCRIPTION-NAME --topic=YOUR-SUBSCRIPTION-NAME --ack-deadline 600

Completing the Pub/Sub Live Indexing Set-Up

Connecting a storage bucket to the Pub/Sub queue must be done using Google’s gsutil command:

gsutil notification create -t YOUR-SUBSCRIPTION-NAME -f json gs://YOUR-BUCKET-NAME

Updated 2 months ago


Live Indexing - Google Pub/Sub


Google PubSub is a fully managed message publish/subscription service that enables you to decouple and scale microservices, distributed systems, and serverless applications.

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.