Terraform for ChaosSearch

Review this topic if you plan to use Terraform to create and manage AWS S3 bucket resources for use with ChaosSearch indexing.

Terraform is an infrastructure service from HashiCorp that enables administrators to configure and provision resources. Terraform has a wide flexibility for implementations, while allows users to design and configure their terraform scripts in a variety of ways.

This topic describes one example of how to use the Terraform commands to perform the following steps:

  • Create an S3 bucket to store data and a role for ChaosSearch to use to read files in that bucket.
  • For Live Indexing support, configure an SQS Queue and Bucket Notification Policy to notify ChaosSearch services when new files are available.
  • For information safety, enable the optional KMS Encryption for the bucket and its contents.

Prerequisites

Before you get started with the Terraform configuration scripts, make sure that you have the following:

  • A ChaosSearch account & external ID
  • An AWS account
  • API keys with admin privileges in the account you want to use to store and index data

ChaosSearch GitHub

GitHub is a website and cloud-based service that helps developers store and manage their code, as well as track and control changes to their code.

ChaosSearch's GitHub repository has several repositories that are being updated with the latest developments for several different use cases. Review the ChaosSearch Github.

Create a New Directory

Terraform uses your working directory for context.

  1. Create a new directory called cs_terraform.
  2. Create a new file called cs_env.tf.
mkdir cs_terraform
cd cs_terraform
touch cs_env.tf
  1. Edit cs_env.tf using a text editor. Type the following information:
    • Type your customer ID and the name of the bucket that you want to create.
    • If you are not using the default region of us-east-1, update the region to the correct value.
provider "aws" {
  alias = "us-east-1"
  region  = "us-east-1"

  version = "~> 2.10"
}

module "s3_customer_buckets" {
  source = "git::https://github.com/ChaosSearch/terraform-modules.git//encrypted-s3-bucket-live-indexing"

  region = "us-east-1"
  cs_external_id = "YOUR_CUSTOMER_ID"
              cs_data_bucket = "YOUR_BUCKET_NAME"
  sqs_queue = true

  providers = {
    aws = "aws.us-east-1"
  }
}

This sample configuration will create several objects:

  • An SQS queue called sqs-s3-YOUR_BUCKET_NAME-us-east-1.
  • A bucket called YOUR_BUCKET_NAME-us-east-1.
  • A bucket notification for YOUR_BUCKET_NAME-us-east-1 that notifies sqs-s3-YOUR_BUCKET_NAME-us-east-1 when new files are created.
  • An IAM role that ChaosSearch can use to access the bucket and the SQS queue.

Initialize

Running terraform init will configure the local Terraform provider, and pull the Terraform module to your local machine.

terraform init
Initializing modules...
- module.s3_customer_buckets

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "aws" (2.10.0)...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Plan

Save the plan to a file called cs_test for additional safety and control. The scenery option helps to format the output to make it easier to read.

terraform plan -out cs_test | scenery

Review the proposed actions of the plan:

  • Create a bucket to hold your data and log files.
+ module.s3_customer_buckets.aws_iam_policy.cs_logging_server_side_role_policy
    id:     
    arn:    
    name:   "cs_logging_server_side_role_policy"
    path:   "/"
    policy: "{
               "Statement": [
                 {
                   "Action": [
                     "s3:Get*",
                     "s3:List*",
                     "s3:PutObjectTagging"
                   ],
                   "Effect": "Allow",
                   "Resource": [
                     "${aws_s3_bucket.cs_data_bucket.arn}",
                     "${aws_s3_bucket.cs_data_bucket.arn}/*"
                   ]
                 },
                 {
                   "Action": [
                     "s3:ListAllMyBuckets"
                   ],
                   "Effect": "Allow",
                   "Resource": "*"
                 },
                 {
                   "Action": "*",
                   "Effect": "Allow",
                   "Resource": [
                     "arn:aws:s3:::cs-${var.cs_external_id}"
                   ]
                 },
                 {
                   "Action": "*",
                   "Effect": "Allow",
                   "Resource": [
                     "arn:aws:s3:::cs-${var.cs_external_id}/*"
                   ]
                 },
                 {
                   "Action": [
                     "kms:GenerateDataKey",
                     "kms:Decrypt"
                   ],
                   "Effect": "Allow",
                   "Resource": [
                     "${aws_kms_key.cs_data_bucket_key.arn}"
                   ]
                 }
               ],
               "Version": "2012-10-17"
             }"

+ module.s3_customer_buckets.aws_iam_role.cs_logging_server_side_role
    id:                    
    arn:                   
    assume_role_policy:    "{
                              "Statement": [
                                {
                                  "Action": "sts:AssumeRole",
                                  "Effect": "Allow",
                                  "Principal": {
                                    "Service": "ec2.amazonaws.com"
                                  }
                                },
                                {
                                  "Action": "sts:AssumeRole",
                                  "Condition": {
                                    "StringEquals": {
                                      "sts:ExternalId": "98ffdf9d-bc56-4943-b89e-113b982e3ec2"
                                    }
                                  },
                                  "Effect": "Allow",
                                  "Principal": {
                                    "AWS": "arn:aws:iam::515570774723:root"
                                  }
                                }
                              ],
                              "Version": "2012-10-17"
                            }"
    create_date:           
    force_detach_policies: "false"
    max_session_duration:  "3600"
    name:                  "cs_logging_server_side_role"
    path:                  "/"
    unique_id:             

+ module.s3_customer_buckets.aws_iam_role_policy_attachment.cs_logging_server_side_role_policy_attach
    id:         
    policy_arn: "${aws_iam_policy.cs_logging_server_side_role_policy.arn}"
    role:       "cs_logging_server_side_role"

+ module.s3_customer_buckets.aws_kms_alias.cs_data_bucket_key
    id:             
    arn:            
    name:           "alias/cs_cs-blog-test_us-east-1"
    target_key_arn: 
    target_key_id:  "${aws_kms_key.cs_data_bucket_key.key_id}"

+ module.s3_customer_buckets.aws_kms_key.cs_data_bucket_key
    id:                      
    arn:                     
    deletion_window_in_days: "10"
    description:             "This key is used to encrypt cs-blog-test-us-east-1"
    enable_key_rotation:     "false"
    is_enabled:              "true"
    key_id:                  
    key_usage:               
    policy:                  

+ module.s3_customer_buckets.aws_s3_bucket.cs_data_bucket
    id:                                                                                                        
    acceleration_status:                                                                                       
    acl:                                                                                                       "private"
    arn:                                                                                                       
    bucket:                                                                                                    "cs-blog-test-us-east-1"
    bucket_domain_name:                                                                                        
    bucket_regional_domain_name:                                                                               
    force_destroy:                                                                                             "false"
    hosted_zone_id:                                                                                            
    lifecycle_rule.#:                                                                                          "1"
    lifecycle_rule.0.abort_incomplete_multipart_upload_days:                                                   "7"
    lifecycle_rule.0.enabled:                                                                                  "true"
    lifecycle_rule.0.expiration.#:                                                                             "1"
    lifecycle_rule.0.expiration.2843080737.date:                                                               ""
    lifecycle_rule.0.expiration.2843080737.days:                                                               "31"
    lifecycle_rule.0.expiration.2843080737.expired_object_delete_marker:                                       ""
    lifecycle_rule.0.id:                                                                                       "cleanup_after_30_days"
    lifecycle_rule.0.transition.#:                                                                             "1"
    lifecycle_rule.0.transition.1899348542.date:                                                               ""
    lifecycle_rule.0.transition.1899348542.days:                                                               "30"
    lifecycle_rule.0.transition.1899348542.storage_class:                                                      "GLACIER"
    region:                                                                                                    
    request_payer:                                                                                             
    server_side_encryption_configuration.#:                                                                    "1"
    server_side_encryption_configuration.0.rule.#:                                                             "1"
    server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.#:                   "1"
    server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.0.kms_master_key_id: "${aws_kms_key.cs_data_bucket_key.arn}"
    server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.0.sse_algorithm:     "aws:kms"
    tags.%:                                                                                                    "1"
    tags.Name:                                                                                                 "cs-blog-test-us-east-1"
    versioning.#:                                                                                              "1"
    versioning.0.enabled:                                                                                      "true"
    versioning.0.mfa_delete:                                                                                   "false"
    website_domain:                                                                                            
    website_endpoint:                                                                                          

+ module.s3_customer_buckets.aws_s3_bucket_notification.cs_data_bucket_notification
    id:                        
    bucket:                    "${aws_s3_bucket.cs_data_bucket.id}"
    queue.#:                   "1"
    queue.0.events.#:          "1"
    queue.0.events.3356830603: "s3:ObjectCreated:*"
    queue.0.id:                
    queue.0.queue_arn:         "${aws_sqs_queue.cs_s3_bucket_sqs.arn}"

+ module.s3_customer_buckets.aws_sqs_queue.cs_s3_bucket_sqs
    id:                                
    arn:                               
    content_based_deduplication:       "false"
    delay_seconds:                     "0"
    fifo_queue:                        "false"
    kms_data_key_reuse_period_seconds: 
    max_message_size:                  "2048"
    message_retention_seconds:         "86400"
    name:                              "s3-sqs-cs-blog-test-us-east-1"
    policy:                            
    receive_wait_time_seconds:         "0"
    tags.%:                            "1"
    tags.Bucket:                       "cs-blog-test-us-east-1"
    visibility_timeout_seconds:        "480"

+ module.s3_customer_buckets.aws_sqs_queue_policy.cs_s3_bucket_sqs
    id:        
    policy:    "{
                  "Statement": [
                    {
                      "Action": "sqs:SendMessage",
                      "Condition": {
                        "ArnEquals": {
                          "aws:SourceArn": "${aws_s3_bucket.cs_data_bucket.arn}"
                        }
                      },
                      "Effect": "Allow",
                      "Principal": "*",
                      "Resource": "${aws_sqs_queue.cs_s3_bucket_sqs.arn}"
                    },
                    {
                      "Action": "sqs:*",
                      "Effect": "Allow",
                      "Principal": {
                        "AWS": [
                          "${aws_iam_role.cs_logging_server_side_role.arn}"
                        ]
                      },
                      "Resource": "${aws_sqs_queue.cs_s3_bucket_sqs.arn}"
                    }
                  ],
                  "Version": "2012-10-17"
                }"
    queue_url: "${aws_sqs_queue.cs_s3_bucket_sqs.id}"

Plan: 9 to add, 0 to change, 0 to destroy.
  • Create a KMS key, and alias, to encrypt your s3 objects at rest.
+ module.s3_customer_buckets.aws_kms_alias.cs_data_bucket_key
+ module.s3_customer_buckets.aws_kms_key.cs_data_bucket_key
  • Create an SQS notification for new objects added to the bucket.
+ module.s3_customer_buckets.aws_s3_bucket_notification.cs_data_bucket_notification
  • Create an SQS Queue for the bucket to send notifications on new objects, and create a user policy to interact with that queue to notify ChaosSearch indexing services when new objects have been added to the bucket.
+ module.s3_customer_buckets.aws_sqs_queue.cs_s3_bucket_sqs
+ module.s3_customer_buckets.aws_sqs_queue_policy.cs_s3_bucket_sqs
  • Create an IAM role that ChaosSearch will use to interact with your bucket using these objects.
+ module.s3_customer_buckets.aws_iam_policy.cs_logging_server_side_role_policy
+ module.s3_customer_buckets.aws_iam_role.cs_logging_server_side_role
+ module.s3_customer_buckets.aws_iam_role_policy_attachment.cs_logging_server_side_role_policy_attach

Apply the Plan

After you review the plan and verify the actions that it will take, apply the plan:

terraform apply cs_test

A successful apply message looks similar to the following. The output can be very long, but the key message is the Apply Complete message at the end.

...
module.s3_customer_buckets.aws_sqs_queue_policy.cs_s3_bucket_sqs: Creation complete after 1s (ID: https://sqs.us-east-1.amazonaws.com/095701894487/s3-sqs-cs-blog-test-us-east-1)
module.s3_customer_buckets.aws_s3_bucket_notification.cs_data_bucket_notification: Creating...
  bucket:                    "" => "cs-blog-test-us-east-1"
  queue.#:                   "" => "1"
  queue.0.events.#:          "" => "1"
  queue.0.events.3356830603: "" => "s3:ObjectCreated:*"
  queue.0.id:                "" => ""
  queue.0.queue_arn:         "" => "arn:aws:sqs:us-east-1:095701894487:s3-sqs-cs-blog-test-us-east-1"
module.s3_customer_buckets.aws_s3_bucket_notification.cs_data_bucket_notification: Creation complete after 1s (ID: cs-blog-test-us-east-1)
  
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

In a few cases, the plan could fail to apply the first time. The typical problem is that it can take some time for AWS to set up resources, and the terraform apply script could time out waiting for resources to become available and known. A sample failure message follows:

Error: Error applying plan:

1 error(s) occurred:
* module.s3_customer_buckets.aws_sqs_queue_policy.cs_s3_bucket_sqs: 1 error(s) occurred

* aws_sqs_queue_policy.cs_s3_bucket_sqs: Error updating SQS attributes: InvalidAttributeValue: Invalid value for the parameter Policy.

status code: 400, request id: c267dace-4173-5178-8ff4-8fae86fb9713

In this case, the common workaround is to run the terraform apply cs_test command again. Usually the apply will complete successfully during the next apply operation. For a subsequent run, the Apply complete message usually reports a smaller number of resources added because it bypasses the already completed work and finishes a shorter list of remaining work, for example:

module.s3_customer_buckets.aws_s3_bucket_notification.cs_data_bucket_notification: Creating...
  bucket:                    "" => "cs-blog-test-us-east-1"
  queue.#:                   "" => "1"
  queue.0.events.#:          "" => "1"
  queue.0.events.3356830603: "" => "s3:ObjectCreated:*"
  queue.0.id:                "" => ""
  queue.0.queue_arn:         "" => "arn:aws:sqs:us-east-1:095701894487:s3-sqs-cs-blog-test-us-east-1"
module.s3_customer_buckets.aws_s3_bucket_notification.cs_data_bucket_notification: Creation complete after 1s (ID: cs-blog-test-us-east-1)

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

More Info: Terraform Blog Post

See Using Terraform to Get Started with ChaosSearch for a blog post of this process.


Did this page help you?