ChaosSearch is designed for easy and secure integration with your cloud-storage solution. ChaosSearch uses read-only access to the buckets where your object files are stored, and read-write access to a new bucket that you own to store the index files for your data. The ChaosSearch Live Indexing feature also uses messaging queue services for notifications when new files are available to index.
Supported cloud storage solutions
Amazon Web Services (AWS) S3
Google Cloud Platform (GCP) Cloud Storage
One or more buckets with the data files to index; ChaosSearch requires read-only access to the object bucket(s).
A new bucket in which ChaosSearch has read-write access for writing the index files created for your objects.
Message queue requirements
For each object group that is configured to use Live Indexing, AWS SQS or GCP Pub/Sub messaging queues are required to send event notifications when new objects are available to index in the read-only bucket(s).
ChaosSearch operates similarly for the supported cloud-storage providers. General references to "cloud storage" refer to any of the supported storage solutions. (When steps or behaviors are specific to one solution, the documentation includes the solution name for clarity.)
There are many ways to configure access to your cloud storage. All providers have IAM control interfaces for roles and policies, and there are often scripted ways as well. See Cloud Platform Prerequisites for an overview of how to configure access to supported storage solutions, either through their application interfaces, or for AWS with scripted AWS CloudFormation or Terraform methods. You can review the required access roles and policies/permissions for the buckets and message services, which can help you to plan for the access to your cloud-storage buckets.
See Pushing Logs to Cloud Storage for some guidance on optimal file size planning and for using various log shippers to land the objects for indexing in your cloud-storage buckets.
Object Storage Considerations
There is no maximum size limit for the objects stored in the cloud-storage buckets. AWS and GCP cloud storage reliability is well suited to support all sizes of files. However, file size could have impacts on performance and data availability. Some up-front planning for optimal object sizes, types, and content can help to improve the overall performance within the cloud resources and ChaosSearch indexing services.
Before you connect to the ChaosSearch console, make sure that you have completed the cloud-storage bucket access setup and you have the following information:
Cloud-Storage and ChaosSearch Access Prerequisites
- An existing AWS or GCP account with bucket privileges
- AWS or GCP account access with read/write IAM privileges
- Your ChaosSearch customer ID for IAM configuration
- ChaosSearch Email / Password
Connect to the ChaosSearch console following the address instructions in your setup email. Log in using your administrator account and password created for you by the ChaosSearch team. Because ChaosSearch is a service, it is easy to get started without days or weeks of software installation and configuration tasks.
When you log in as an administrator, the system usually displays the Storage view with a list of cloud-storage buckets and any object groups that were created. (Some users might be configured to open other pages like Analytics.)
For new setups, you might have to configure your bucket access credentials to see your storage buckets. To check and provide credentials, click your account name in the top right corner to display the tasks menu, and open the Settings/Help page:
The Settings/Help page has menus that you can use to change your account password, specify credentials for your AWS or GCP storage, display the API keys, or access help. A sample AWS Credentials page follows.
Copy and paste the AWS ARN (or the GCP Service Account ID) into the credentials field to enable ChaosSearch read-access to your storage buckets. If you are not sure how to obtain those service credentials, the instructions are in Cloud Platform Prerequisites.
After the access information for your cloud storage is configured, Storage shows the buckets found in your cloud storage location, as in the screen shown at the start of this topic.
There are many buckets in the list, but you can use only the bucket(s) associated with your cloud-storage credentials. If you select a bucket that you are not permitted to use, the system displays a message:
The buckets list has a filter field to help reduce the list to only those bucket names that contain the string that you type.
The ellipses menu (three dots icon after the filter field) has options that you can use to sort the bucket list, refresh the bucket list, select all object groups (to see a list of the existing groups in the main panel), and options to delete or import a bucket. See Storage Overview for more details on how to use the buckets list and the features.
When you select a bucket that contains data that you want to index, the window updates to show the files and folders within the bucket. You can use the File View area to navigate to the files that you want to index.
Updated 6 months ago
After you have configured access to your buckets, create one or more object groups to index the files for visualization.