Live Indexing - Amazon SQS

How to enable AWS SQS messaging for Live Indexing of objects in AWS S3 cloud storage buckets.

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications.

216216

​What does ChaosSearch offer AWS SQS users?

ChaosSearch Live Indexing supports an integration with Amazon SQS event notifications. With the SQS Integration configured, ChaosSearch is notified about new objects written to the S3 storage buckets associated with an object group.

With Live Indexing enabled for an object group, ChaosSearch can monitor your S3 bucket for object creation events via user-specified Amazon SQS Queue and index the newly-created objects, automatically making the new data available for querying.

Add SQS Integration in ChaosSearch - Quick Option

🚧

AWS IAM Role

Please note that the AWS IAM Role configured for the ChaosSearch account must also be configured with appropriate access to the SQS Queue.

Add SQS Integration in ChaosSearch

To enable SQS support for your S3 bucket:

  1. Log in to the AWS console using your AWS account.
  2. In the AWS Management Console, navigate to the Simple Queue Service -> Create Queue page.
21842184
  1. In the Name field, type a name for the queue.
  2. In the Visibility timeout field, increase the default value from from 30 seconds to 5 minutes.
  3. Scroll down to Access Policy.
22082208
  1. Select the Advanced option.
  2. In the JSON window, replace the default content with the following JSON statements that ensure that the S3 bucket can send events to the queue and that ChaosSearch has permission to see messages on the queue.

Make sure that you update the AWS Id value, the new-sqs-queue-name, and desired-bucket-name-here statements below with your AWS ID, the SQS queue name, the target bucket name, and the role ARN for your AWS account.

{
  "Version": "2012-10-17",
  "Id": "arn:aws:sqs:us-east-1:111111111111:new-sqs-queue-name",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:us-east-1:111111111111:new-sqs-queue-name",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:s3:::desired-bucket-name-here"
        }
      }
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111111111111:role/chaossearch-role"
      },
      "Action": "sqs:*",
      "Resource": "arn:aws:sqs:us-east-1:11111111111:new-sqs-queue-name"
    }
  ]
}
  1. Click Create queue. The following page appears to confirm that the queue is created.
27762776

You must add the SQS policy to the AWS user account that you use to access the bucket.

  1. Log in to the AWS Console and navigate to the IAM section.
  2. On the Roles page, locate the role that you created for your ChaosSearch access.
29522952
  1. Click Attach policies. A sample window follows (your AWS role name could be different).
18341834
  1. Filter to display the SQS policy that you created, select the policy, and click Attach policy. The Roles page appears and shows the SQS policy attached to the role.
19141914

When you create the object group for the bucket that has been updated with the SQS messaging controls, the Live Indexing option requires the ARN for your SQS Policy. You can obtain the ARN by clicking your SQS policy name and copying the ARN value for your configuration. A sample follows:

13821382

Adding SQS ARN to ChaosSearch Object Group

You must associate your SQS Policy ARN value with the object group that you create to index the related S3 bucket.

  1. Create your object group following the steps described in Creating object groups.
  2. In the Create Object Group window, as in the following example, paste the SQS policy ARN value from AWS into the Live Indexing field. A sample follows.
11481148
  1. Click Create. The object group is created.
  2. Navigate to the object group Group Contents tab, and you should see that Live Indexing is configured. A sample window follows.
  3. Click Start Indexing when you are ready to index the related objects in the S3 bucket.
20462046

Completing the SQS Live Indexing Setup

  1. Navigate to the S3 console.
  2. Click on the name of the S3 bucket associated with your Live Indexing object group. The S3 bucket details page opens.
  3. Click Properties.
  4. Scroll down in the page to the Events Notifications section.
  5. Click Create event notification. The Create event notification page appears.
  6. Under General configuration, in the Event name field, type a name.
  7. Under Event types, Object creation, select All object create events.
15001500
  1. Scroll down to Destination, select SQS Queue as the destination, then select the SQS policy that you associated.
14781478

Notes

Some important notes:

  • For indexing efficiency, ChaosSearch uses a hold-down time (approximately 2-3 minutes) to batch indexing data, so there will be a delay before indexed data is available for query.
  • Live Indexing does not update the ChaosSearch index files when a source data file is deleted. If a data file is deleted after it has been indexed, the content from the data file remains in the existing index files.

Did this page help you?