bucket

Yes

The name of the object group to create. The name must begin and end with a letter or number. Bucket names can contain only lowercase letters, numbers, and hyphens (-).

public, type, content_type, source_id

No

These are fields used by ChaosSearch teams; do not use.

source

Yes

The name of the cloud storage bucket where the objects for the group are stored

format

No

A configuration block with file format details

type

No

The type of the files in the group. Valid values are CSV, JSON, LOG, or Unknown.

column_delimiter

No

For CSV files, the column separator character

header_row

No

For CSV files, a boolean value to indicate whether the files include a header row before the data

row_delimiter

No

The row separator character

pattern

No

For LOG files, the regular expression pattern to use to parse the fields of the log file. ChaosSearch has several default regex patterns to parse common or previously seen log files. You can specify your own regex for your custom logs or new types of log files.

array_flatten_depth

No

For JSON files, a numeric value that specifies how deeply to flatten the properties contained in nested arrays at levels 1-10, all, or 0 to flatten no arrays. A value of -1 is unlimited flattening. Any arrays that are not flattened are indexed and stored as JSON strings, so their data is still captured within the Chaos Index content.

strip_prefix

No

A value of false causes all JSON file fields to be prefixed with the root. characters. If set to true, which is recommended, the root. prefix is stripped from the group's field names.

Up to provider v1.0.11, the default is false. Starting with provider v1.0.12, the default is true.

Note that the Create Object Group UI strips the root. prefix by default.

horizontal

No

For JSON files, a boolean value that specifies whether to index the JSON file using horizontal flattening/expansion. The default (false) is to use vertical flattening.

array_selection

No

For JSON files, a jsonencode value that specifies the arrays that you want to index as fields (or to exclude and index only as concatenated JSON string values). For more information on JSON array indexing, see JSON Array Options.

field_selection

No

For JSON files, a jsonencode value that specifies JSON nested objects that you might want to index as JSON strings (rather than as separate fields). For more information on field selection and rules, see Field Inclusion and Exclusion Policies.

vertical_selection

No

For JSON files, a jsonencode value that specifies the arrays that you want to vertically expand when an object group uses horizontal expansion by default. For more information on JSON array indexing, see JSON Array Options.

live_events

No

The SQS ARN to associate with the group when creating a Live Index group

index_retention

No

The amount of time in days to retain the indexed data for the object group. This value can be changed for an existing object group.

overall

Specify a value in days. A value of -1 is unlimited retention (old data will not be deleted). The default for a new group is 14 days.

filter

Yes

A configuration block containing filtering rules to specify the bucket objects to include in the group based on object name prefixes or object pathname patterns.

NOTE: The prefix, regex and equals filters must all be specified in their own filter block.

field

Specifies what the filter applies to. Valid values are key and storageClass.

A value of key indicates that the object key (which is usually the object name in cloud storage) is the match value for the prefix.

range

Specifies a min and max value for the filtering range.

prefix

Yes

Used with the key field, this is the prefix string to use as the matching value for selecting objects to include in the group. A value of foo/ will match objects with names that begin with foo.

regex

No

Used with the key field. Specifies a regex string to use as the matching value for selecting objects to include in the group. A value of .* will match all objects. A value of ​App1\/events.*\.csv matches any events\*.csv files in the App1 folder.

equals

No

Used with the storageClass field. Specifies the storageClass type of the S3 bucket. Valid values are STANDARD, STANDARD_IA, INTELLIGENT_TIERING, ONEZONE_IA, GLACIER, DEEP_ARCHIVE, or REDUCED_REDUNDANCY.

options

No

A configuration block to specify advanced settings

compression

No

Specifies the compression used for the files indexed by the object group. Valid values are GZIP or SNAPPY.

col_types

No

Overrides a column's data type. For example, you could set a number type used as an ID to be treated as a string. Example:\

col_types = jsonencode({  
      "TimeStamp": "Timeval"  
    })

col_renames

No

Overrides a column's name. For example, you could change a column name to match a well-known name for your users. Example:\

col_rename = jsonencode({  
      "TimeStamp": "Period"  
    })

col_selection

No

Specify the fields to include (or exclude) as part of the object group index. By default, all object file columns are included as object group fields.

col_selection has the includes (whitelist) and excludes (blacklist) forms, where you must specify the exact field names to match in the source file. The regex for with its include true/false setting uses one or more regex patterns to identify the fields to include or exclude in the index. You can specify one or more col_selection policies, where multiple policies are an AND condition, which means that a column must match all specified policies to qualify.
Examples:

   col_selection = jsonencode([{  
      "includes": [  
        "foo","bar"  
      ],  
      "type": "whitelist"  
    }])

   col_selection = jsonencode([{  
      "include": false
      "patterns": [  
        "^source$","^host$"  
      ],  
      "type": "regex"  
    }])

partition_by

No

Specifies a regular expression to use as an isolation key for the object group.