Create a new object group

post

https://example.com/Bucket/createObjectGroup

Create a new object group that is unique to the ChaosSearch service. The Bucket endpoints currently use AWS Signature Version 4 authentication.

To list or delete existing object groups, use the S3-style endpoints (/V1/).

Body Params

bucket

string

required

Specifies the object group name.

Object group names can be any string without forward slash characters (path separators). Object group names must be unique. Because object groups are modeled in ChaosSearch as an S3 bucket, like views, this name must be unique across any bucket name.

Example:

"my-object-group-0"

For best compatibility, names should comply with S3 bucket name restrictions:

Names must be between 3 and 63 characters long.
Names can contain only lowercase letters, numbers, dots (.), and hyphens (-).
Names must begin and end with a letter or number.
Names must not be formatted as an IP address (for example, 192.168.5.4).
Names cannot begin with xn-- (for buckets created after February 2020).

NOTE: Internal bucket resources use a prefix of cs or system.. To prevent naming conflicts, do not name buckets with either prefix.

source

string

required

The name of the cloud storage bucket that backs this object group.

Example:

"my-bucket"

filter

array of objects

required

An object group filter is a predicate that filters the objects (data files) that will be associated with and indexed by this group.

Example:

"filter": [{"field":"key","regex":".*"}]

This example filter configures the system to apply the regex .* to the S3 keys of the objects encountered by indexing. It will consider all objects in the associated cloud storage bucket when indexing.

Example:

"filter": {
  "AND": [
    {"field": "key","prefix": "foo/"},
    {"field": "key","regex": ".*"}
  ]
}

This example adds a prefix predicate. The predicates are joined with a logical AND to produce the filter. As with the first example, the filter is applied to each object key of the object group. The filter considers only objects that have a prefix of foo.

Example:

"filter": {
  "OR": [
    {"field": "key","prefix": "foo/"},
    {"field": "key","regex": ".*"}
  ]
}

This example extends the previous example so that objects that have either the prefix of 'foo/' OR the key of '.*' will be indexed.

filter*

format

object

required

options

object

required

liveEvents

string

The ARN of the AWS SQS or GCP Pub/Sub queue that receives bucket event messages; ChaosSearch uses these events to index objects as they are written.

Configuration of the SQS/PubSub queue and bucket event notifications are done via the AWS APIs.

Queues are not validated for correctness (such as correct permissions, or queue exists).

When supplied, the object group created is considered a live index, otherwise it is a static index.

interval

object

The index interval (daily) is enabled by passing a value of 0 for the mode and column. Otherwise there will be no index interval (a single file will be produced).

NOTE: To disable the interval do not supply this key.

indexRetention

Defaults to 14

The ChaosSearch retention policy allows you to specify how long to retain data for querying. During indexing, ChaosSearch writes an index file per day of ingested data. These daily indexes can be set to expire over time by specifying an overall retention policy.

The default is 14 days for new object groups.

NOTE: daily intervals MUST be configured for the object group to use index retention.

partitionBy

string

A regular expression with capture groups used to isolate the indexes of the object group.

The regex is applied to each cloud storage object when indexing. Isolated slices are created by taking the resulting strings from the capture groups. When no capture groups are matched for a given key, the default key is used.

Objects in the bucket that parse will be indexed regardless of a match against the partitionBy regex.

Example:

# Files on S3
1. /587656/blue/2020-10-15.json
2. /587656/blue/2020-10-16.json
3. /587656/green/2020-10-15.json
4. /587656/green/2020-10-16.json
5. /foo/blue/2020-10-15.json
6. /2020-10-16.json

"partitionBy": {
  "by": [
    {
      "_type": "chaossumo.service.cs3.model.DataSet.PartitionBy.ByKey",
      "pattern": "/([0-9]+)/(.*)/.*"
    }
  ]
}

The regex generates isolation slices of "587656" and "blue" for the
first two files, "587656" and "green" for the second two files, "blue"
for the fifth file, and the default slice for the sixth.

overwrite

boolean

Responses

200

successful creation

400The specified resource already exists

500The input was invalid