The Data Lake Engine for Scalable Log Analysis

Welcome to ChaosSearch. Store Everything. Ask Anything.

This section walks through the high-level capabilities of the ChaosSearch platform. Here you can find information on the Storage, Refinery, and Analytics page along with our introductory video.

ChaosSearch is a fully managed, secure service on AWS backed by S3 as a data store. With a few clicks, customers can be up and running in minutes, all at a fraction of the cost of running your own Elasticsearch cluster or ELK Stack.

Learn how ChaosSearch unlocks your Amazon S3 storage and turns it into a secure, durable, and cost-effective search platform with both Amazon S3 and Elasticsearch APIs.

Before you dive into the platform, we want you to understand the core concepts of the platform. This will help you understand all the other extra features and capabilities to best achieve your use cases, and save you time during initial configurations.

What is ChaosSearch

Step 1: Create IAM Role & Policy

The settings section is the best starting point, here you will start the process of finalizing the IAM role provisioning and begin inviting members of your team. Other sections in the Settings part of the platform will help you as you continue to build out your logging strategy.

At ChaosSearch, we give you the ability to make more of your S3 infrastructure by turning into an elastic cluster. The first step in doing so is following the recommended set-up to start the creation of your IAM Role and Policy which will allow us to start listing out all of the S3 buckets within your AWS (Amazon Web Services) account.

Step 2: Discovery

Discovery is the first step in understanding all of the files that are stored in your S3 bucket(s). When Discovery starts you will begin to see the different fields populate and update as we continue to understand the contents of the bucket. Items we will list out are:

  • Total # of Files
  • Total Size of Bucket
  • Creation Date
  • File Type
  • Security Statistics
  • Trending Prefixes

Step 3: Object Groups

Object Groups are customizable filters for viewing what’s in your buckets for fine-grained object analysis. These virtual buckets are our first steps in auto-discovering and indexing your data. As you move through the Object Group creation you will have the opportunity to define what filtering is needed to separate out the different files. Object Groups can be used for building filters on microservices, log type, log per application, etc.

Once you’ve defined the filter, we will see that our Data Format and Compression has already been recognized for us. If this has not happened, from the drop down you will have a chance to change any unrecognized formats and compression. At this point, we can configure if we want to set-up our live indexing for this object group or what indexing intervals we would like to have in place.

Step 4: Indexing

In order to see the platform in action, start indexing the data in one of your S3 buckets. During the indexing process, you will start to see different stats populating in the Group Details, Index Details, and Indexed Structure.

The Indexed Structure field will represent all of the mappings the ChaosSearch platform has done and will show the Types of each field.

Step 5: Visualization

Now that your data has been indexed, it is available to Elasticsearch and Kibana. Navigate to the Visualization tab to get started. In order to visualize and explore data in Kibana, you'll need to create an index pattern to retrieve data from Elasticsearch.

The Visualize screen allows you to create a variety of charts and graphs fueled by your ChaosSearch indices.

Review some of our Use Cases

Updated 8 months ago

The Data Lake Engine for Scalable Log Analysis

Welcome to ChaosSearch. Store Everything. Ask Anything.

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.