DocumentationChaosSearch Blog
Documentation
StatusHome Page

Real-Time Indexing - Google Pub/Sub

Google PubSub is a fully managed message publish/subscription service that enables you to decouple and scale microservices, distributed systems, and serverless applications.

Suggest Edits
174

What does ChaosSearch offer GCP PubSub users?

ChaosSearch provides the ability for live indexing with the integration for Google Pub/Sub. With the Pub/Sub Integration configured, ChaosSearch will be able to identify new files that have been put into the queue for discovery and indexing.

Functionality of Live Indexing

With Live Indexing enabled for an Object Group, ChaosSearch will monitor your GCP storage bucket for object creation events via user-specified PubSub subscription and index the newly-created objects, making the data available for querying as it becomes available.

Add PubSub Integration in ChaosSearch - Quick Option

🚧

GCP Service account

Please note that the GCP Service account configured for the ChaosSearch account must also be configured with appropriate access to the Pub/Sub subscription.

Add Pub/Sub Integration in ChaosSearch

  1. In the GCP Management Console - Navigate to PubSub -> Create Topic
  2. Assign the topic a name - do not create a default subscription
536
  1. Click “Subscriptions” on the left menu, and create a subscription with the same name as your topic - increasing the Acknowledgement deadline to 600 seconds:
945

Alternately, you can do the above with the following command using Google’s gcloud utility:

gcloud --project YOUR-PROJECT-ID pubsub topics create YOUR-TOPIC-NAME
gcloud --project YOUR-PROJECT-ID pubsub subscriptions create YOUR-SUBSCRIPTION-NAME --topic=YOUR-TOPIC-NAME --ack-deadline 600

Adding Pub/Sub Subscription to ChaosSearch Object Group

  1. Create an object group
  2. Enter your Project ID and PubSub Subscription ID into the appropriate fields
  3. Click the Realtime box
  4. Click Create
  5. Click Start Indexing
568
  1. Navigate back to the Object Group you will now see Live Indexing configured
925

Completing the Pub/Sub Live Indexing Set-Up

Connecting a storage bucket to the Pub/Sub queue must be done using Google’s gsutil command:

gsutil notification create -f json -p YOUR-LOGS-PREFIX -e OBJECT_FINALIZE -t YOUR-TOPIC-NAME gs://YOUR-BUCKET-NAME

Updated about 1 year ago