ChaosSearch API Overview

ChaosSearch APIs give you interconnectivity to process your requests and access data.

API Overview

The ChaosSearch API is the administrative interface for the ChaosSearch service. ChaosSearch admin API was originally modeled as an extension to the S3 API. Most of the API calls, parameters, etc will look familiar to those who created and managed S3 buckets.

Create Object Group

Path: /Bucket/createObjectGroup

Summary: Creates a new Object Group
description: |
Creates a new Object Group which is unique to the ChaosSearch service.
        
In order to get a listing or delete Object Groups an S3 style endpoint is used (e.g. /V1/).
        
The update API (see /Bucket/updateObjectGroup) is limited to a subset of fields, so it will be necessary to delete and create an Object Group in order to update all fields.
      requestBody:
        content:
          application/json:
            schema:
              type: object
              properties:
                bucket:
                  type: string
                  description: |
                    Object Group name.
                    
Object Group names can be any string without forward slash characters (path separators) Object Group names must be unique. Because Object Groups are modeled in ChaosSearch as an S3 Bucket, as are Views, this name must be unique across any Bucket name.

Example
     "my-object-group-01"
                    
For best compatibility names should comply with S3 bucket name restrictions:
    * Bucket names must be between 3 and 63 characters long.
    * Bucket names can consist only of lowercase letters, numbers, dots (.), and hyphens (-).
    * Bucket names must begin and end with a letter or number.
    * Bucket names must not be formatted as an IP address (for example, 192.168.5.4).
    * Bucket names can't begin with xn-- (for buckets created after February 2020).
    * Bucket names must be unique within a partition. A partition is a grouping of Regions.

AWS currently has three partitions: aws (Standard Regions), aws-cn (China Regions), and aws-us-gov (AWS GovCloud [US] Regions).
    * Buckets used with Amazon S3 Transfer Acceleration can't have dots (.) in their names.

For more information about transfer acceleration, see Amazon S3 Transfer Acceleration.
    source:
        type: string
        description: The name of the S3 Bucket that backs this Object Group.
            example: my-bucket
                filter:
            type: array
        items:
              $ref: '#/components/schemas/ObjectGroupFilter'
        
        description:
         An Object Group filter is a predicate used to filter Objects that will be indexed by the system.
                    
Example
 "filter": [{"field":"key","regex":".*"}]
                    
The above example filter instructs the system to apply the regex ".*" to the S3 keys of the Objects encountered by indexing. It will consider all Objects of the given Object Group when indexing.
                    
Example
 "filter":{"AND": [{"field": "key","prefix": "foo/"},{"field": "key","regex": ".*"}]}
                    
This example adds a prefix predicate. The predicates will be joined with a logical AND to produce the filter. As with the first example, it will be applied to each Object key of the Object Group. It will consider only Objects that have a prefix of "foo".

Example
  "filter":{"OR": [{"field": "key","prefix": "foo/"},{"field": "key","regex": ".*"}]}

An example that adjusts the previous example so Objects that either have the prefix of 'foo/' OR the key of '.*' will be accepted.

format:
    $ref: '#/components/schemas/Format'
    options:
    type: object
    properties:
    ignoreIrregular:
    type: boolean
    description:
        Ignore Irregular Objects while indexing a bucket (deprecated)
                        
This setting should always be set `true`.
    compression:
    type: string
  enum:
        - GZIP
        - ZLIB
        - SNAPPY
    description: |
        Compression algorithm used to compress the Object stored in S3, used by ChaosSearch to decompress the archive for indexing.
                        
NOTE: Do not supply this key if the Objects in the bucket are not compressed
        columnTypes:
        type: object
        description: |
        Column Types can be used to override a given column's data type.
        Parsing of Objects will produce values of the overridden type
                        
Example
    colTypes": { "foo": "NUMBER" }
                        
                        The above example will override the data type of the column named "foo" so its value will be a NUMBER
                      example: { "foo": "NUMBER" }
                  required:
                    - ignoreIrregular
                liveEvents:
                  type: string
                  example: 'arn:aws:sqs:us-east-1:0123456789012:s3-notifications'
                  description: |
                    the ARN of the SQS queue that receives S3 Bucket event messages used by
                    ChaosSearch to index Objects as they are written.
                    
                    Configuration of the SQS queue and bucket event notifications are done via
                    the AWS APIs.
                    
                    Queues are not validated for correctness (i.e. correct permissions, queue
                    exists, etc)
                    
                    When supplied the Object Group created is considered a Live index, otherwise it will be a static index.
                interval:
                  type: object
                  properties:
                    mode:
                      type: integer
                    column:
                      type: integer
                  required:
                    - mode
                    - column
                  description: |
                    The index interval (daily) may be enabled by passing a value of 0 for the
                    mode and column. Otherwise there will be no index interval (a single file
                    will be produced)
                    
                    NOTE: To disable the interval do not supply this key.
                  example: { "mode": 0, "column": 0 }
                indexRetention:
                  allOf:
                    - $ref: '#/components/schemas/TimeWindow'
                    - description: The number of days data will be retained before it's deleted. To disable the retention policy (and keep data indefinitey) do NOT supply this key
                partitionBy:
                  type: string
                  example: "regex-with-(capture-group1)-(capture-group2)"
                  description: |
                    a regular expression with capture groups used to partition the indexes of the Object Group.
                    
                    The regex is applied to each S3 key encountered by indexing. Partitions are created by taking the resulting strings from the capture groups. When no capture groups are matched for a given key the default partition is used.
                    
                    Objects in the bucket that parse will be indexed regardless of a match against the partitionBy regex.
                    
                    Example:
                    
                        # Files on S3
                        1. /587656/blue/2020-10-15.json
                        2. /587656/blue/2020-10-16.json
                        3. /587656/green/2020-10-15.json
                        4. /587656/green/2020-10-16.json
                        5. /foo/blue/2020-10-15.json
                        6. /2020-10-16.json
                        
                        partitionBy: "/([0-9]+)/(.*)/.*"
                        
                        The regex will generate a partition of "587656" and "blue" for the
                        first two files, "587656" and "green" for the second two files, "blue"
                        for the fifth file, and the default partition for the sixth.
              required:
                - bucket
                - source
                - filter
                - contentType
                - format
                - options
              example:
                bucket: my-object-group
                source: my-s3-bucket
                format:
                  _type: "CSV"
                  horizontal: true
                  stripPrefix: false
                interval:
                  mode: 0
                  column: 0
                indexRetention: 14
                filter:
                  field: key 
                  regex: '.*'
                options:
                  ignoreIrregular: false
                  compression: GZIP
      responses:
        '200':
          description: successful creation
        '400':
          $ref: '#/components/responses/DuplicateBucketError'
        '500':
          $ref: '#/components/responses/InvalidInputError'

Update Object Group

Path: /Bucket/updateObjectGroup

summary: Updates an existing Object Group
      description: Updates an existing Object Group
      requestBody:
        content:
          application/json:
            schema:
              type: object
              properties:
                bucket:
                  type: string
                  description: Object Group Name
                indexRetention:
                  $ref: '#/components/schemas/TimeWindow'
                targetActiveIndex:
                  default: 1
                  type: integer
                  description: |
                    Allow overriding number of "irrevokable" leases on compute for tasks associated with in. Use with care.
                    
                    Object Group indexing model uses a lease system to allocate workers (registered units) for indexing jobs. Each worker is allowed one
                    lease. By default an index for a given Object Group will require 1 lease in order to make progress. By setting the target active index
                    value to something larger than 1, the system will attempt to allocate more leases for indexing. If the system has no more leases to give,
                    the index will progress as long as it has at least 1 lease allocated.
                    
                    By default the value is 1
                    
                    ChaosSearch support should be enagaged to help tune this value for latency sensitive indexes.
              required:
                - bucket
      responses:
        '200':
          description: Successful update of existing Object Group
        '400':
          $ref: '#/components/responses/DuplicateBucketError'
        '500':
          $ref: '#/components/responses/InvalidInputError'

Create View

Path: /Bucket/createView

summary: Creates a new View
      description: |
        Creates a new View. A view is a virtual Object Group, comprising 1 or more Object Groups.
        
        Views add permissions, default time fields, etc
        
        To list or delete Views, an S3 style endpoint is used (e.g. /V1/).
        
        Note: The update API is limited to a subset of fields, necessary to delete and create a View in order to update all fields.
      requestBody:
        content:
          application/json:
            schema:
              type: object
              properties:
                bucket:
                  type: string
                  description: View Name
                sources:
                  type: array
                  items:
                    type: string
                  description: |
                    The View's source of data, a list of Object Group Names.
                    
                    IMPORTANT: An empty array means ALL Object Groups!
                    
                    Note: View Name cannot be identical with an Object Group Name.
                indexPattern:
                  type: string
                  description: |
                    An Index Pattern (regex) that matches and filters source based on index names, practically speaking from Object Groups Names.
                    
                    Example:
                    
                        Object Group Name: foo
                        Resulting index names (for daily indexes):
                        _foo_2020-10-15_
                        _foo_2020-10-16_
                        
                        Resulting index names:
                        _foo_
                caseInsensitive:
                  type: boolean
                  default: false
                  description: |
                    Flag to toggle case-sensitivity of searches of all values in the View
                    
                    Example:
                    
                        row: {"Foo": "Bar"}
                        Searching for "Bar" is treated case-insensitive, but keys such as "Foo" are still case-sensitive.
                indexRetention:
                  $ref: '#/components/schemas/ViewTimeWindow'
                transforms:
                  type: array
                  default: []
                  items:
                    $ref: '#/components/schemas/Transform'
                  description: transformations on fields used to produce new fields from existing ones
                filters:
                  description: |
                    TODO describe what and how filters work, especially predicates
                    
                    Also, how to add AND, and OR logic here
                    
                    Example log:
                    
                        {
                          "system": "foo",
                          "type": "elb-access",
                          "stage": "staging",
                        }
                        
                        User allowed to see all data where "system" = "foo", and "type" matches ".*(-access$)" but not
                        "stage" = "production".
                    
                    Example:
                    
                        "filter": {
                          "predicate": {
                              "field": "attrs.Cluster",
                              "query": "au-production",
                              "state": {
                                  "_type": "chaossumo.query.QEP.Predicate.TextMatchState.Exact"
                              },
                              "_type": "chaossumo.query.NIRFrontend.Request.Predicate.TextMatch"
                          }
                        }
              required:
                - bucket
                - sources
                - indexPattern
      responses:
        '200':
          description: Successful creation of View
        '400':
          $ref: '#/components/responses/DuplicateBucketError'
        '500':
          $ref: '#/components/responses/InvalidInputError'

Update View

Path: /Bucket/updateView

summary: Updates an existing View
      description: Updates an existing View
      requestBody:
        content:
          application/json:
            schema:
              type: object
              properties:
                bucket:
                  type: string
                  description: View Name
                indexRetention:
                  $ref: '#/components/schemas/ViewTimeWindow'
      responses:
        '200':
          description: Successful update of existing View
        '400':
          $ref: '#/components/responses/DuplicateBucketError'
        '500':
          $ref: '#/components/responses/InvalidInputError'
  /V1/:
    get:
      description: Lists the existing Object Groups.
      responses:
        '200':
          $ref: '#/components/responses/SuccessfulListing'
  /V1/{key}:
    delete:
      description: Delete Views/Object Groups
      parameters:
        - name: key
          description: |
            {key} can be a View Name, or an Object Group Name
          in: path
          schema:
            type: string
          required: true
      responses:
        '200':
          description: Successful delete of Object Group
        '400':
          description: The specified resource does not exist

Search API

MSearch API

ChaosSearch primarily uses the _msearch ES API to support Kibana running on data in the ChaosSearch platform. Here is an example way to get a few of your queries accomplished over _msearch:

Example Python Script

from elasticsearch import Elasticsearch, helpers, exceptions, RequestsHttpConnection
import json
from requests_aws4auth import AWS4Auth

awsauth = AWS4Auth("Access-Key-ID", "Secret-Access-Key", "us-east-1", 's3')
es = Elasticsearch(
  hosts = [{'host': 'poc-trial.chaossearch.io', 'port': 443, 'url_prefix': '/elastic', 'use_ssl': True}],
  http_auth=awsauth,
  connection_class=RequestsHttpConnection,
  verify_certs=True
)

client = es

try:
    client_info = Elasticsearch.info(client)
    print ('Elasticsearch client info:', json.dumps(client_info, indent=4))
except exceptions.ConnectionError as err:
    print ('Elasticsearch client error:', err)
    client = None
if client != None:
    search_body = {
        "size": 500,
        "query": {
            "match_all": {}
        }
    }

    resp = helpers.scan(
        client,
        index = 'Index-View-Name', # <---- SPECIFY HERE
        query = search_body,
        size = 12,
    )

    print (type(resp))

    for num, doc in enumerate(resp):
        print ('\n', num, '', doc)

Updated about a month ago

ChaosSearch API Overview


ChaosSearch APIs give you interconnectivity to process your requests and access data.

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.