Step 1. Enable Access to Cloud Storage

ChaosSearch is designed for easy and secure integration with your cloud-storage solution. ChaosSearch uses read-only access to the buckets where your object files are stored, and read-write access to a new bucket that you own to store the index files for your data. The ChaosSearch Live Indexing feature also uses messaging queue services for notifications when new files are available to index.

👍

ChaosSearch operates similarly for the supported cloud-storage providers.

General references to "cloud storage" refer to any of the supported storage solutions. (When steps or behaviors are specific to one solution, the documentation includes that specific cloud solution name for clarity.)

There are many ways to configure access to your cloud storage. All providers have IAM control interfaces for roles and policies, and there are some scripted access provisioning methods as well. See Configure Cloud Storage Access for an overview of how to configure access to supported storage solutions, either through their application interfaces, or for AWS with scripted AWS CloudFormation or Terraform methods. You can review the required access roles and policies/permissions for the buckets and message services, which can help you to plan for the access to your cloud-storage buckets.

See Pushing Logs to Cloud Storage for some guidance on optimal file size planning and for using various log shippers to land the objects for indexing in your cloud-storage buckets.

👍

Object Storage Considerations

There is no maximum size limit for the objects stored in the cloud-storage buckets. AWS and GCP cloud storage reliability is well suited to support all sizes of files. File size does matter for performance and data availability. Some up-front planning for optimal object sizes, types, and content can help to improve the overall performance within the cloud resources and ChaosSearch indexing services.

Creating the ChaosSearch Resources

ChaosSearch Customer Success and the Platform team work with you to size and create the ChaosSearch compute environment that includes the workers to index your data and to run the services for your account. ChaosSearch teams manage those resources including performing sizing and upgrading tasks when needed for the account. Free trial accounts have an initial (smaller) ChaosSearch configuration, and proof-of-value (POV) users and customers typically have a larger configuration that is based on their ingest and usage rates.

As a result of the configuration, your ChaosSearch console is set up but might not yet be provisioned with access to your storage buckets. You will have a domain name, tenant account, and a ChaosSearch external ID for the cluster of resources.

Verify Access Prerequisites

Before you connect to the ChaosSearch console, make sure that you have completed the cloud-storage bucket access setup and you have the following information:

👍

Cloud-Storage and ChaosSearch Access Prerequisites

• Your existing AWS or GCP account with bucket privileges
• AWS or GCP account access with read/write IAM privileges
• Your ChaosSearch customer ID for IAM configuration
• ChaosSearch Email / Password

Log in to ChaosSearch

If you're the primary user at your site, connect to the ChaosSearch console following the address instructions in your setup email. Log in using your primary/administrator account and password created for you by the ChaosSearch team. Because ChaosSearch is a service, it is easy to get started without days or weeks of software installation and configuration tasks.

When you log in as an administrator, the system usually displays the Storage area with a list of cloud-storage buckets and any object groups that have been created. (Some non-admin users could be configured to open other pages like Search Analytics.)

👍

Survey the contents of your cloud storage buckets before you start creating object groups. A high-level inventory of the storage file types, how files are organized in folders, identifying any data or security concerns for the files, identifying content that should be omitted from indexes, and the volume and frequency of file updates can all help with object group planning.

Are Your Buckets Missing? Check the Credentials

If you do not see your buckets (or any buckets) on the Storage page, check and set (or correct) your bucket access credentials. To check and provide credentials, click your account name in the top right corner to display the tasks menu, and open the Settings/Help page:

The Settings/Help page has menus that you can use to specify credentials for your AWS or GCP storage. A sample AWS Credentials page follows.

Copy and paste the AWS ARN (or the GCP Service Account ID on the GCP Credentials page) to enable ChaosSearch read-access to your storage buckets. If you are not sure how to obtain those service credentials, see the instructions in Configure Cloud Storage Access.

Buckets

When the access information for your cloud storage is configured, Storage shows the buckets found in your cloud storage location.

There could be many buckets in the list, but you can access only the bucket(s) associated with your cloud-storage credentials. If you select a bucket that you are not permitted to use, the system blocks access and displays a message:

The buckets list in the left pane has a search field to narrow down the list to only those bucket and object group names that contain a string that you type.

The ellipses menu (three dots icon after the filter field) has options to sort the bucket list, refresh the list, select all groups, show/hide hidden buckets, or to import a bucket. See Storage Overview for more details on how to use the buckets menu and the features.

When you select a bucket that contains the data that you want to index, the window updates to show the files and folders within the bucket. You can use the file view area and its search filter controls to find to the files that you want to index.