Step 1. Enable Access to Cloud Storage
If you already have a data lake in cloud storage, or if you are just getting started, read how to connect ChaosSearch to your storage buckets and objects.
ChaosSearch is designed for easy and secure integration with your cloud-storage solution. ChaosSearch uses read-only access to the buckets where your object files are stored, and read-write access to a new bucket that you own to store the index files for your data. The ChaosSearch Live Indexing feature also uses messaging queue services for notifications when new files are available to index.
| Requirement | Description |
|---|---|
| Supported cloud storage solutions | Amazon Web Services (AWS) S3 Google Cloud Platform (GCP) Cloud Storage |
| Bucket requirements | One or more buckets with the data files to index; ChaosSearch requires read-only access to the log and event object bucket(s). A new bucket in which ChaosSearch has read-write access for storing the index files for your objects and configuration information. |
| Message queue requirements | For each object group that uses Live Indexing, an AWS SQS or GCP Pub/Sub messaging queue is required to send event notifications when new log and event file objects become available to index in the customer read-only bucket(s). |
ChaosSearch operates similarly for the supported cloud-storage providers. General references to "cloud storage" refer to any of the supported storage solutions. (When steps or behaviors are specific to one solution, the documentation includes that specific cloud solution name for clarity.)
There are many ways to configure access to your cloud storage. All providers have IAM control interfaces for roles and policies, and there are often scripted APIs as well. See Cloud Platform Prerequisites for an overview of how to configure access to supported storage solutions, either through their application interfaces, or for AWS with scripted AWS CloudFormation or Terraform methods. You can review the required access roles and policies/permissions for the buckets and message services, which can help you to plan for the access to your cloud-storage buckets.
See Pushing Logs to Cloud Storage for some guidance on optimal file size planning and for using various log shippers to land the objects for indexing in your cloud-storage buckets.
Object Storage Considerations
There is no maximum size limit for the objects stored in the cloud-storage buckets. AWS and GCP cloud storage reliability is well suited to support all sizes of files. File size can have impacts on performance and data availability. Some up-front planning for optimal object sizes, types, and content can help to improve the overall performance within the cloud resources and ChaosSearch indexing services.
Verify Access Prerequisites
Before you connect to the ChaosSearch console, make sure that you have completed the cloud-storage bucket access setup and you have the following information:
Cloud-Storage and ChaosSearch Access Prerequisites
- An existing AWS or GCP account with bucket privileges
- AWS or GCP account access with read/write IAM privileges
- Your ChaosSearch customer ID for IAM configuration
- ChaosSearch Email / Password
Log in to ChaosSearch
If you're the primary user at your site, connect to the ChaosSearch console following the address instructions in your setup email. Log in using your primary/administrator account and password created for you by the ChaosSearch team. Because ChaosSearch is a service, it is easy to get started without days or weeks of software installation and configuration tasks.
When you log in as an administrator, the system usually displays the Storage area with a list of cloud-storage buckets and any object groups that have been created. (Some non-admin users could be configured to open other pages like Search Analytics.)
Survey the contents of your cloud storage buckets before you start creating object groups. A high-level inventory of the storage file types, how files are organized in folders, identifying any data or security concerns for the files, identifying content that should be omitted from indexes, and the volume and frequency of file updates can all help with object group planning.
Are Your Buckets Missing? Check the Credentials
If you do not see your buckets (or any buckets) on the Storage page, check and set (or correct) your bucket access credentials. To check and provide credentials, click your account name in the top right corner to display the tasks menu, and open the Settings/Help page:
The Settings/Help page has menus that you can use to specify credentials for your AWS or GCP storage. A sample AWS Credentials page follows.
Copy and paste the AWS ARN (or the GCP Service Account ID on the GCP Credentials page) to enable ChaosSearch read-access to your storage buckets. If you are not sure how to obtain those service credentials, see the instructions in Cloud Platform Prerequisites.
Buckets
When the access information for your cloud storage is configured, Storage shows the buckets found in your cloud storage location.
There could be many buckets in the list, but you can access only the bucket(s) associated with your cloud-storage credentials. If you select a bucket that you are not permitted to use, the system blocks access and displays a message:
The buckets list in the left pane has a search field to narrow down the list to only those bucket and object group names that contain a string that you type.
The ellipses menu (three dots icon after the filter field) has options that you can use to sort the bucket list in alphanumerical order or reverse order, or refresh the bucket list. See Storage Overview for more details on how to use the buckets list and the features.
When you select a bucket that contains the data that you want to index, the window updates to show the files and folders within the bucket. You can use the file view area and its search filter controls to find to the files that you want to index.
Updated 10 days ago
After you have configured access to your buckets, create one or more object groups to index the files for visualization.