Live Indexing - Amazon SQS

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications.

โ€‹What does ChaosSearch offer AWS SQS users?

ChaosSearch provides the ability for live indexing with the integration for Amazon SQS. With the SQS Integration configured, ChaosSearch will be able to identify new files that have been put into the queue for discovery and indexing.

Functionality of Live Indexing

  • With Live Indexing enabled for an Object Group, ChaosSearch will monitor your S3 bucket for object creation events via user-specified Amazon SQS Queue and index the newly-created objects, making the data available for querying as it becomes available.

Add SQS Integration in CHAOSSEARCH - Quick Option

๐Ÿšง

AWS IAM Role

Please note that the AWS IAM Role configured for the ChaosSearch account must also be configured with appropriate access to the SQS Queue.

Add SQS Integration in CHAOSSEARCH

  1. In the AWS Management Console - Navigate to the Simple Queue Service -> Create Queue
  2. Assign the Queue a name and increase the visibility timeout from 30 seconds to 5 minutes.
  1. Once the SQS Queue is configured - Navigate to IAM section within the AWS Console
  2. Select the IAM Role -> Policies and click "Create policy"
  3. Select the JSON tab and input the below JSON with your new SQS ARN into the Policy field:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:DeleteMessage",
        "sqs:DeleteMessageBatch",
        "sqs:SendMessage",
        "sqs:ReceiveMessage",
        "sqs:GetQueueUrl"
      ],
      "Resource": [
        "arn:aws:sqs:us-east-1:123456789:cs-us-1-east-logs"
      ]
    }
  ]
}
  1. Click Review Policy
  2. Enter the custom policy name into policy type filter
  3. Attach the policy to the previously created ChaosSearchIAM Role
  4. Check the custom policy and select Next: Review
  5. Copy the SQS ARN for later use in the ChaosSearch configuration

Adding SQS ARN to ChaosSearch Object Group

  1. Create an Creating object groups
  2. Copy and Paste the SQS ARN into the Live Indexing field
  3. Click Create
  • Navigate back to the Object Group you will now see Live Indexing configured

Completing the SQS Live Indexing Set-Up

  • Navigate to the S3 console
  • Select the appropriate S3 bucket
  • Click Properties -> Scroll Down to the Events section
  • Select the SQS Queue from the Drop

AWS SQS CloudFormation Template

---
AWSTemplateFormatVersion: '2010-09-09'
Description: Chaossearch Live Indexing Pipeline
Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
    - Label:
        default: Chaossearch Pipeline Creation
      Parameters:
      - ChaossearchQueueName
    ParametersLabels:
      ChaossearchQueueName:
        default: Name your Chaossearch SQS Queue
Parameters:
  ChaossearchQueueName:
    Description: Input the name of the SQS Queue
    Type: String
Resources:
  ChaossearchQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName:
        Ref: ChaossearchQueueName
      VisibilityTimeout: 300
  ChaossearchQueuePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      PolicyDocument:
        Id: CSQueuePolicy
        Version: '2012-10-17'
        Statement:
        - Sid: Allow-CS-SendMessage
          Effect: Allow
          Principal: "*"
          Action: sqs:SendMessage
          Resource: "*"
      Queues:
      - Ref: ChaossearchQueue
Outputs:
  SQSSource:
    Description: The name of the Chaossearch SQS Queue
    Value: 
      Fn::GetAtt:
      - ChaossearchQueue
      - QueueName

๐Ÿ“˜

Stack Creation Complete

Once the CloudFormation stack has completed, you will need to add the Event Notification to the SQS Queue

Notes

  • For Indexing efficiency, ChaosSearch has introduced a hold-down time (approximately 2-3 minutes) to batch indexing data, so there will be a delay before indexed data is available for query.

  • Live Indexing does not consider file deletion. If a file is deleted and it was already indexed, it will continue to be included in the resulting index.

Updated 23 days ago

Live Indexing - Amazon SQS


Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications.

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.