The Data Lake Platform for Analytics at Scale

Know Better™. Now you can, while realizing the true promise of data lake economics for log analytics at scale, insightful product-led growth and agile BI.

This section walks through the high-level capabilities of the ChaosSearch platform. Here you can find information on the Storage, Refinery, and Analytics page along with our introductory video.

ChaosSearch is a fully managed, secure service on AWS/GCP backed by Cloud object storage as a data store. With a few clicks, customers can be up and running in minutes, all at a fraction of the cost of running your own Elasticsearch cluster or ELK Stack.

Learn how ChaosSearch unlocks your Cloud object storage and turns it into a secure, durable, and cost-effective search platform with both Cloud object storage and Elasticsearch APIs.

Before you dive into the platform, we want you to understand the core concepts of the platform. This will help you understand all the other extra features and capabilities to best achieve your use cases, and save you time during initial configurations.

What is ChaosSearch

Step 1: Create IAM Role & Policy or GCP Service Accounts

The settings section is the best starting point, here you will start the process of finalizing the service access to your Cloud object storage and begin inviting members of your team. Other sections in the Settings part of the platform will help you as you continue to build out your logging strategy.

At ChaosSearch, we give you the ability to make more of your Cloud object storage infrastructure by turning into an elastic cluster. The first step in doing so is following the recommended set-up to start the creation of your IAM Role and Policy which will allow us to start listing out all of the Cloud object storage buckets within your AWS (Amazon Web Services) / GCP (Google Cloud Platform) account.

Step 2: Discovery

Discovery is the first step in understanding all of the files that are stored in your Cloud object storage bucket(s). When Discovery starts you will begin to see the different fields populate and update as we continue to understand the contents of the bucket. Items we will list out are:

  • Total # of Files
  • Total Size of Bucket
  • Creation Date
  • File Type
  • Security Statistics
  • Trending Prefixes

Step 3: Object Groups

Object Groups are customizable filters for viewing what’s in your buckets for fine-grained object analysis. These virtual buckets are our first steps in auto-discovering and indexing your data. As you move through the Object Group creation you will have the opportunity to define what filtering is needed to separate out the different files. Object Groups can be used for building filters on microservices, log type, log per application, etc.

Once you’ve defined the filter, we will see that our Data Format and Compression has already been recognized for us. If this has not happened, from the drop down you will have a chance to change any unrecognized formats and compression. At this point, we can configure if we want to set-up our live indexing for this object group or what indexing intervals we would like to have in place.

Step 4: Indexing

In order to see the platform in action, start indexing the data in one of your Cloud object storage buckets. During the indexing process, you will start to see different stats populating in the Group Details, Index Details, and Indexed Structure.

The Indexed Structure field will represent all of the mappings the ChaosSearch platform has done and will show the Types of each field.

Step 5: Visualization

Now that your data has been indexed, it is available to Elasticsearch and Kibana. Navigate to the Visualization tab to get started. In order to visualize and explore data in Kibana, you'll need to create an index pattern to retrieve data from Elasticsearch.

The Visualize screen allows you to create a variety of charts and graphs fueled by your ChaosSearch indices.

Review some of our Use Cases

Updated 2 months ago


The Data Lake Platform for Analytics at Scale


Know Better™. Now you can, while realizing the true promise of data lake economics for log analytics at scale, insightful product-led growth and agile BI.

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.