Field Selection

ChaosSearch offers administrators several policy and processing options to control how to index their source file data. The following table lists the field selection policy types. More information is available after the table.

📘

NOTE:

The Field Selection user interface requires you to create and import a JSON file that contains the settings for the field selections and processing rules. That JSON file must be stored in a local file location that you can access and import from the browser.

Applying a Field Selection Filter

To specify the field selection filter for an object group:

1. In the object group content preview window, click Field Overrides in the top right corner.

2. In the Field Overrides window, go to the Field Selection area at the bottom.

3. Click the Drop JSON files here to upload link to open a file browser, and select the JSON file with the field selection rules. Alternatively, you could drag and drop the JSON file into the area.
4. Click Save to add the field selection rules to the object group.

📘

The Save button appears to hang if the JSON file has errors.

When you create the JSON file for the field selection policies that you want to use, make sure that you verify that you have a properly configured JSON file. It can be helpful to validate the file using a third-party or public JSON validator. If you click Save and the window appears to hang, the problem is usually a formatting error in the JSON file. Verify the file, fix any format errors if needed, and save the JSON file.

When the object group is created, ChaosSearch applies the field selection rules to the resulting fields in the index schema. You must index the object group to see the effect of your policies on the resulting fields. It can be helpful to index a small set of files and to review the group's Properties information to confirm that the policies are taking effect as expected.

Field Inclusion and Exclusion Policies

As a default behavior, ChaosSearch indexes all of the fields in source log and event files that match the object group content filters. Sometimes log and event files contain fields that are not meaningful for analytics or queries. Rather than index information that is not valuable, it is a good practice to exclude unnecessary fields.

👍

Use field inclusion/exclusion to manage the object group indexed fields without transforming raw files.

While it is a common practice to remove undesired content in the log creation phase (before files are written to cloud storage), that content cleanup is not always possible or easy. Changing the log content can be complex, require iterations, and could become a repetitive task. Fields deemed unnecessary for one form of analysis might be useful for another, adding duplication and rework to get the right content into the logs.

The ChaosSearch inclusion and exclusion indexing policies can help to filter the log and event source fields for an object group to only what is needed, reducing or eliminating the need for costly rework of the source files. The resulting ChaosSearch indexed data for the group is more tailored and compact, leading to better query performance and end-user information experience.

The field selection whitelist and blacklist policies define the rules for including or excluding fields that exist within the source log files.

If most of the source file fields are useful for indexing and analysis, you could use a blacklist policy to exclude specific fields from the indexed data. The blacklist can contain one or more fields, and the index process will skip (not index) those fields. Similarly, if the source log and event files have only a subset of fields that are valuable for queries, you could use a whitelist to index only those fields and omit all others in the indexed data for that object group.

A sample blacklist definition contains statements similar to the following. In this example, the referenced fields (field_name and field_name2) of the source files will not be included as fields in the indexed data. All other fields will be included/indexed.

{
  "type": "blacklist",
  "excludes": [ "field_name", "field_name2", ...  ]
}

A whitelist definition for specifying a list of fields to include/index contains statements similar to the following. Only the referenced fields in the whitelist (field_name and field_name2) will be included in the object group and indexed data. All other fields in the source log and event files will be omitted from the object group index.

{
  "type": "whitelist",
  "includes": [ "field_name", "field_name2", … ]
}

There is no limit to the number of fields that you can include or exclude.

For JSON files, make sure that you capture the full and correct field name, which might require a property,prop1[.prop2...] pathname. it can be helpful to create a test object group for a small sample file to correctly identify the field names in a sample resulting object group to ensure that you have the correct field names.

JSON Array Options

For JSON files with nested arrays, you can use the JSON Flex options to specify how to index the arrays and their nested properties. For example, you can use horizontal or vertical expansion, and you can specify an array level (that is, how many nested levels) of properties to index as separate fields. See JSON Flex Options Reference for more information about JSON files and the available ChaosSearch optimizations for JSON file indexing.

When JSON files have nested arrays and are indexed using horizontal flattening, you can use the vertical_selection_policy to specify one or more array fields to vertically expand to use them for analytics and filtering controls within searches and visualizations.

The vertical_selection_policy options support includes/whitelist and excludes/blacklist syntax to control which arrays are or are not vertically expanded. Use the blacklist options when you want to exclude a nested array within a vertically expanded array from vertical expansion.

A sample JSON vertical_selection_policy definition follows, where an array called app_status will be vertically expanded during indexing:

{"vertical_selection_policy":[{"includes":["app-status"],"type":"whitelist"}]}

For JSON files that are indexed vertically or when you use the vertical_selection_policy to make nested arrays fully indexed and available for analytics and filter controls, you can use the array_selection_policy to specify one or more arrays to ignore for vertical expansion; that is, the specified array and its entire contents will be stored as a JSON string in the indexed field. The JSON string content is searchable for analytics and querying.

A sample array_selection_policy follows, where the Records.requestParameters array will be indexed as a JSON string, not as individual fields and rows.

{"array_selection_policy":[{"excludes":["Records.requestParameters"],"type":"blacklist"}]}

As a combination example, there might be cases where you want to use horizontal flattening in general, but vertical expansion for a specific array, and then JSON string indexing for a specific nested subproperty of a vertically-expanded array. With JSON Flex, you can layer the options for this indexing behavior. The following example vertically expands the Records array and the Records.requestParameters.eventInfo array, and indexes the specific nested arrays Records.requestParameters.HistoricalMetrics and Records.responseElements.MetricResults as JSON strings.

{
 "vertical_selection_policy": [
  {
   "includes": [
    "Records",
    "Records.requestParameters.eventInfo"
   ],
   "type": "whitelist"
  }
 ],
 "array_selection_policy": [
  {
   "excludes": [
    "Records.requestParameters.HistoricalMetrics",
    "Records.responseElements.MetricResults"
   ],
   "type": "blacklist"
  }
 ]
}

JSON Nested Object Options

Nested objects are indexed as separate properties, which could result in a large number of separate fields in the indexed data. Some nested objects and properties might be better suited and more efficient to index as JSON string blobs rather than as separate property fields.

The field_selection_policy allows you to specify the names of nested JSON objects that you want to index as a JSON string blob instead of separate properties. The contents of that nested object and any nested properties below it are indexed as one contiguous JSON string field. The JSON string content will be text searchable like any string field, with the benefit of a reduced number of indexed data fields in the object group.

The field_selection_policy options support an include/regex and excludes/blacklist syntax to specify the nested objects. You can use include with a regex to use regular expressions for matching the nested fields to process as a JSON string. You can also use a blacklist to specify the nested field names to index as JSON strings.

Sample field selection policy definitions follow. The first example uses regex patterns to specify the fields to treat as JSON strings. Examples for a blacklist rule or a combination of both forms are on separate tabs.

"field_selection_policy": [
    {
      "include": false,
      "patterns": [
        "^field[.]prop[.][A-Fa-f0-9]{64}.*",
        "^property[.][A-Fa-f0-9]{64}.*"
      ],
      "type": "regex"
    }
]

  "field_selection_policy": [
    {
      "excludes": [
        "plan.jobs",
        "unusable_job_outputs"
      ],
      "type": "blacklist"
    }
  ]

{
  "field_selection_policy": [
    {
      "include": false,
      "patterns": [
        "Records[.]userI.*[.]sessionContext[.]attr.*",
        "Records[.]req.*"
      ],
      "type": "regex"
    },
    {
      "excludes": [
        "Records.additionalEventData"
      ],
      "type": "blacklist"
    }     
  ]
}

👍

Materialize with JSON

The JSON fields and properties that are converted to JSON string blobs normally cannot be used for analytics as filters or metrics for aggregations, however ChaosSearch offers a Refinery view technique to materialize a property inside a JSON blob for analytics use.

When you create the Refinery view, you can use the Materialize with JSON transform to specify an embedded JSON property by its JSON path, and the property will become available for analytics and filtering as a post-analytics, schema-on-read, materialized column.