Live Indexing - Amazon SQS

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications.

​What does ChaosSearch offer AWS SQS users?

ChaosSearch provides the ability for live indexing with the integration for Amazon SQS. With the SQS Integration configured, ChaosSearch will be able to identify new files that have been put into the queue for discovery and indexing.

Functionality of Live Indexing

  • With Live Indexing enabled for an Object Group, ChaosSearch will monitor your S3 bucket for object creation events via user-specified Amazon SQS Queue and index the newly-created objects, making the data available for querying as it becomes available.

Add SQS Integration in ChaosSearch - Quick Option

🚧

AWS IAM Role

Please note that the AWS IAM Role configured for the ChaosSearch account must also be configured with appropriate access to the SQS Queue.

Add SQS Integration in ChaosSearch

  1. In the AWS Management Console - Navigate to the Simple Queue Service -> Create Queue
  2. Assign the Queue a name and increase the visibility timeout from 30 seconds to 5 minutes.
  1. Scroll down to Access Policy, select advanced, then in the json editor insert the policy listed to below.

This need to ensure the S3 bucket can send events to the queue AND ChaosSearch has the rights to see messages on the queue.

📘

Please make sure you update policy with your AWS ID, Resource, and SourceArn

Here is an example policy:

{
  "Version": "2012-10-17",
  "Id": "arn:aws:sqs:us-east-1:111111111111:new-sqs-queue-name",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:us-east-1:111111111111:new-sqs-queue-name",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:s3:::desired-bucket-name-here"
        }
      }
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111111111111:role/chaossearch-role"
      },
      "Action": "sqs:*",
      "Resource": "arn:aws:sqs:us-east-1:11111111111:new-sqs-queue-name"
    }
  ]
}
  1. Navigate to IAM section within the AWS Console - Select the IAM Role -> Policies and click "Create policy"
  2. Select the JSON tab and input the below JSON with your new SQS ARN into the Policy field:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:DeleteMessage",
        "sqs:SendMessage",
        "sqs:ReceiveMessage",
        "sqs:GetQueueUrl"
      ],
      "Resource": [
        "arn:aws:sqs:us-east-1:123456789:cs-us-1-east-logs"
      ]
    }
  ]
}
  1. Click Review Policy
  2. Enter the custom policy name into policy type filter
  3. Attach the policy to the previously created ChaosSearchIAM Role
  4. Check the custom policy and select Next: Review
  5. Copy the SQS ARN for later use in the ChaosSearch configuration

Adding SQS ARN to ChaosSearch Object Group

  1. Create an Creating object groups
  2. Copy and Paste the SQS ARN into the Live Indexing field
  3. Click Create
  4. Click Start Indexing
  • Navigate back to the Object Group you will now see Live Indexing configured

Completing the SQS Live Indexing Set-Up

  • Navigate to the S3 console
  • Select the appropriate S3 bucket
  • Click Properties -> Scroll Down to the Events Notifications section
  • Name the notification, click All object create events, Scroll down to Destinations then select the SQS Queue as the destination, then select the desired SQS Queue

Notes

  • For Indexing efficiency, ChaosSearch has introduced a hold-down time (approximately 2-3 minutes) to batch indexing data, so there will be a delay before indexed data is available for query.

  • Live Indexing does not consider file deletion. If a file is deleted and it was already indexed, it will continue to be included in the resulting index.

Updated 2 months ago


Live Indexing - Amazon SQS


Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications.

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.