Terraform for ChaosSearch
Review this topic if you plan to use Terraform to create and manage AWS S3 bucket resources for use with ChaosSearch indexing.
Terraform is an infrastructure service from HashiCorp that enables administrators to configure and provision resources. Terraform has a wide flexibility for implementations, while allows users to design and configure their terraform scripts in a variety of ways.
This topic describes one example of how to use the Terraform commands to perform the following steps:
- Create an S3 bucket to store data and a role for ChaosSearch to use to read files in that bucket.
- For Live Indexing support, configure an SQS Queue and Bucket Notification Policy to notify ChaosSearch services when new files are available.
- For information safety, enable the optional KMS Encryption for the bucket and its contents.
Prerequisites
Before you get started with the Terraform configuration scripts, make sure that you have the following:
- A ChaosSearch account & external ID
- An AWS account
- API keys with admin privileges in the account you want to use to store and index data
ChaosSearch GitHub
GitHub is a website and cloud-based service that helps developers store and manage their code, as well as track and control changes to their code.
ChaosSearch's GitHub repository has several repositories that are being updated with the latest developments for several different use cases. Review the ChaosSearch Github.
Create a New Directory
Terraform uses your working directory for context.
- Create a new directory called
cs_terraform. - Create a new file called
cs_env.tf.
mkdir cs_terraform
cd cs_terraform
touch cs_env.tf
- Edit
cs_env.tfusing a text editor. Type the following information:- Type your customer ID and the name of the bucket that you want to create.
- If you are not using the default region of us-east-1, update the region to the correct value.
provider "aws" {
alias = "us-east-1"
region = "us-east-1"
version = "~> 2.10"
}
module "s3_customer_buckets" {
source = "git::https://github.com/ChaosSearch/terraform-modules.git//encrypted-s3-bucket-live-indexing"
region = "us-east-1"
cs_external_id = "YOUR_CUSTOMER_ID"
cs_data_bucket = "YOUR_BUCKET_NAME"
sqs_queue = true
providers = {
aws = "aws.us-east-1"
}
}
This sample configuration will create several objects:
- An SQS queue called
sqs-s3-YOUR_BUCKET_NAME-us-east-1. - A bucket called
YOUR_BUCKET_NAME-us-east-1. - A bucket notification for
YOUR_BUCKET_NAME-us-east-1that notifiessqs-s3-YOUR_BUCKET_NAME-us-east-1when new files are created. - An IAM role that ChaosSearch can use to access the bucket and the SQS queue.
Initialize
Running terraform init will configure the local Terraform provider, and pull the Terraform module to your local machine.
terraform init
Initializing modules...
- module.s3_customer_buckets
Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "aws" (2.10.0)...
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Plan
Save the plan to a file called cs_test for additional safety and control. The scenery option helps to format the output to make it easier to read.
terraform plan -out cs_test | scenery
Review the proposed actions of the plan:
- Create a bucket to hold your data and log files.
+ module.s3_customer_buckets.aws_iam_policy.cs_logging_server_side_role_policy
id:
arn:
name: "cs_logging_server_side_role_policy"
path: "/"
policy: "{
"Statement": [
{
"Action": [
"s3:Get*",
"s3:List*",
"s3:PutObjectTagging"
],
"Effect": "Allow",
"Resource": [
"${aws_s3_bucket.cs_data_bucket.arn}",
"${aws_s3_bucket.cs_data_bucket.arn}/*"
]
},
{
"Action": [
"s3:ListAllMyBuckets"
],
"Effect": "Allow",
"Resource": "*"
},
{
"Action": "*",
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::cs-${var.cs_external_id}"
]
},
{
"Action": "*",
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::cs-${var.cs_external_id}/*"
]
},
{
"Action": [
"kms:GenerateDataKey",
"kms:Decrypt"
],
"Effect": "Allow",
"Resource": [
"${aws_kms_key.cs_data_bucket_key.arn}"
]
}
],
"Version": "2012-10-17"
}"
+ module.s3_customer_buckets.aws_iam_role.cs_logging_server_side_role
id:
arn:
assume_role_policy: "{
"Statement": [
{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
}
},
{
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "98ffdf9d-bc56-4943-b89e-113b982e3ec2"
}
},
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::515570774723:root"
}
}
],
"Version": "2012-10-17"
}"
create_date:
force_detach_policies: "false"
max_session_duration: "3600"
name: "cs_logging_server_side_role"
path: "/"
unique_id:
+ module.s3_customer_buckets.aws_iam_role_policy_attachment.cs_logging_server_side_role_policy_attach
id:
policy_arn: "${aws_iam_policy.cs_logging_server_side_role_policy.arn}"
role: "cs_logging_server_side_role"
+ module.s3_customer_buckets.aws_kms_alias.cs_data_bucket_key
id:
arn:
name: "alias/cs_cs-blog-test_us-east-1"
target_key_arn:
target_key_id: "${aws_kms_key.cs_data_bucket_key.key_id}"
+ module.s3_customer_buckets.aws_kms_key.cs_data_bucket_key
id:
arn:
deletion_window_in_days: "10"
description: "This key is used to encrypt cs-blog-test-us-east-1"
enable_key_rotation: "false"
is_enabled: "true"
key_id:
key_usage:
policy:
+ module.s3_customer_buckets.aws_s3_bucket.cs_data_bucket
id:
acceleration_status:
acl: "private"
arn:
bucket: "cs-blog-test-us-east-1"
bucket_domain_name:
bucket_regional_domain_name:
force_destroy: "false"
hosted_zone_id:
lifecycle_rule.#: "1"
lifecycle_rule.0.abort_incomplete_multipart_upload_days: "7"
lifecycle_rule.0.enabled: "true"
lifecycle_rule.0.expiration.#: "1"
lifecycle_rule.0.expiration.2843080737.date: ""
lifecycle_rule.0.expiration.2843080737.days: "31"
lifecycle_rule.0.expiration.2843080737.expired_object_delete_marker: ""
lifecycle_rule.0.id: "cleanup_after_30_days"
lifecycle_rule.0.transition.#: "1"
lifecycle_rule.0.transition.1899348542.date: ""
lifecycle_rule.0.transition.1899348542.days: "30"
lifecycle_rule.0.transition.1899348542.storage_class: "GLACIER"
region:
request_payer:
server_side_encryption_configuration.#: "1"
server_side_encryption_configuration.0.rule.#: "1"
server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.#: "1"
server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.0.kms_master_key_id: "${aws_kms_key.cs_data_bucket_key.arn}"
server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.0.sse_algorithm: "aws:kms"
tags.%: "1"
tags.Name: "cs-blog-test-us-east-1"
versioning.#: "1"
versioning.0.enabled: "true"
versioning.0.mfa_delete: "false"
website_domain:
website_endpoint:
+ module.s3_customer_buckets.aws_s3_bucket_notification.cs_data_bucket_notification
id:
bucket: "${aws_s3_bucket.cs_data_bucket.id}"
queue.#: "1"
queue.0.events.#: "1"
queue.0.events.3356830603: "s3:ObjectCreated:*"
queue.0.id:
queue.0.queue_arn: "${aws_sqs_queue.cs_s3_bucket_sqs.arn}"
+ module.s3_customer_buckets.aws_sqs_queue.cs_s3_bucket_sqs
id:
arn:
content_based_deduplication: "false"
delay_seconds: "0"
fifo_queue: "false"
kms_data_key_reuse_period_seconds:
max_message_size: "2048"
message_retention_seconds: "86400"
name: "s3-sqs-cs-blog-test-us-east-1"
policy:
receive_wait_time_seconds: "0"
tags.%: "1"
tags.Bucket: "cs-blog-test-us-east-1"
visibility_timeout_seconds: "480"
+ module.s3_customer_buckets.aws_sqs_queue_policy.cs_s3_bucket_sqs
id:
policy: "{
"Statement": [
{
"Action": "sqs:SendMessage",
"Condition": {
"ArnEquals": {
"aws:SourceArn": "${aws_s3_bucket.cs_data_bucket.arn}"
}
},
"Effect": "Allow",
"Principal": "*",
"Resource": "${aws_sqs_queue.cs_s3_bucket_sqs.arn}"
},
{
"Action": "sqs:*",
"Effect": "Allow",
"Principal": {
"AWS": [
"${aws_iam_role.cs_logging_server_side_role.arn}"
]
},
"Resource": "${aws_sqs_queue.cs_s3_bucket_sqs.arn}"
}
],
"Version": "2012-10-17"
}"
queue_url: "${aws_sqs_queue.cs_s3_bucket_sqs.id}"
Plan: 9 to add, 0 to change, 0 to destroy.
- Create a KMS key, and alias, to encrypt your s3 objects at rest.
+ module.s3_customer_buckets.aws_kms_alias.cs_data_bucket_key
+ module.s3_customer_buckets.aws_kms_key.cs_data_bucket_key
- Create an SQS notification for new objects added to the bucket.
+ module.s3_customer_buckets.aws_s3_bucket_notification.cs_data_bucket_notification
- Create an SQS Queue for the bucket to send notifications on new objects, and create a user policy to interact with that queue to notify ChaosSearch indexing services when new objects have been added to the bucket.
+ module.s3_customer_buckets.aws_sqs_queue.cs_s3_bucket_sqs
+ module.s3_customer_buckets.aws_sqs_queue_policy.cs_s3_bucket_sqs
- Create an IAM role that ChaosSearch will use to interact with your bucket using these objects.
+ module.s3_customer_buckets.aws_iam_policy.cs_logging_server_side_role_policy
+ module.s3_customer_buckets.aws_iam_role.cs_logging_server_side_role
+ module.s3_customer_buckets.aws_iam_role_policy_attachment.cs_logging_server_side_role_policy_attach
Apply the Plan
After you review the plan and verify the actions that it will take, apply the plan:
terraform apply cs_test
A successful apply message looks similar to the following. The output can be very long, but the key message is the Apply Complete message at the end.
...
module.s3_customer_buckets.aws_sqs_queue_policy.cs_s3_bucket_sqs: Creation complete after 1s (ID: https://sqs.us-east-1.amazonaws.com/095701894487/s3-sqs-cs-blog-test-us-east-1)
module.s3_customer_buckets.aws_s3_bucket_notification.cs_data_bucket_notification: Creating...
bucket: "" => "cs-blog-test-us-east-1"
queue.#: "" => "1"
queue.0.events.#: "" => "1"
queue.0.events.3356830603: "" => "s3:ObjectCreated:*"
queue.0.id: "" => ""
queue.0.queue_arn: "" => "arn:aws:sqs:us-east-1:095701894487:s3-sqs-cs-blog-test-us-east-1"
module.s3_customer_buckets.aws_s3_bucket_notification.cs_data_bucket_notification: Creation complete after 1s (ID: cs-blog-test-us-east-1)
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
In a few cases, the plan could fail to apply the first time. The typical problem is that it can take some time for AWS to set up resources, and the terraform apply script could time out waiting for resources to become available and known. A sample failure message follows:
Error: Error applying plan:
1 error(s) occurred:
* module.s3_customer_buckets.aws_sqs_queue_policy.cs_s3_bucket_sqs: 1 error(s) occurred
* aws_sqs_queue_policy.cs_s3_bucket_sqs: Error updating SQS attributes: InvalidAttributeValue: Invalid value for the parameter Policy.
status code: 400, request id: c267dace-4173-5178-8ff4-8fae86fb9713
In this case, the common workaround is to run the terraform apply cs_test command again. Usually the apply will complete successfully during the next apply operation. For a subsequent run, the Apply complete message usually reports a smaller number of resources added because it bypasses the already completed work and finishes a shorter list of remaining work, for example:
module.s3_customer_buckets.aws_s3_bucket_notification.cs_data_bucket_notification: Creating...
bucket: "" => "cs-blog-test-us-east-1"
queue.#: "" => "1"
queue.0.events.#: "" => "1"
queue.0.events.3356830603: "" => "s3:ObjectCreated:*"
queue.0.id: "" => ""
queue.0.queue_arn: "" => "arn:aws:sqs:us-east-1:095701894487:s3-sqs-cs-blog-test-us-east-1"
module.s3_customer_buckets.aws_s3_bucket_notification.cs_data_bucket_notification: Creation complete after 1s (ID: cs-blog-test-us-east-1)
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
More Info: Terraform Blog Post
See Using Terraform to Get Started with ChaosSearch for a blog post of this process.
Updated 5 months ago