A Closer Look at JSON Object Group Settings

Review the settings used for JSON object groups to see the flexible indexing options.

When you create an object group in ChaosSearch Storage, and you select one or more JSON files to index, the object group Content Preview window displays information and indexing options. A sample Content Preview window follows:

As shown in the highlighted summary row:

  • The Format of the selected sample file is JSON.

  • Compression shows whether the JSON file is not compressed (NONE), or if GZIP or SNAPPY compression is detected.

  • The Array Flatten Depth controls how deeply into nested arrays you want to be able to vertically expand.

Depth Setting

Description

UNLIMITED

The default of UNLIMITED specifies that all nested arrays are expanded to their constituent attributes and values. This type of data is typically a better option for horizontal expansion.

NONE

A depth of NONE specifies that all nested arrays are stored as a JSON text blob. The values cannot be expanded for analysis by individual attributes; this option supports full text search queries within the JSON string. This can be a common setting and could be helpful in cases where JSON files are only used for string searches and no attribute filtering is needed.

1–10

A number from 1-10 specifies that arrays at or above the specified level can be expanded and used for analytics, but any lower-level arrays are saved as native JSON strings and cannot be expanded. A value of 2 specifies that arrays at level 1 and level 2 can be expanded, but arrays at levels 3 and lower cannot.

🚧

Important:

Unlimited array flattening could trigger a permutation explosion (and storage impact) for certain complex JSON files, especially those with arrays of arrays. As a best practice, carefully review the JSON file structure, the location of the analysis data of interest, and how analysts plan to query or visualize that data. This can help to determine a suitable flattening depth for the JSON files and avoid unnecessary storage and indexing impacts.

📘

Expansion

When data "cannot be expanded," the impact is that the data is stored with all the JSON properties in the array concatenated as one continuous string. This supports simple SQL-style string searches for content inside the array/column. Filters or queries on specific array attribute/value combinations are not possible with JSON blobs.

  • Expansion is the flattening method, either Vertical or Horizontal.

Expansion Option

Description

Vertical

Vertical expansion is a flattening that typically benefits analytics by creating an index record for each JSON array member and nested array (depending on the flatten depth). The resulting index supports more flexibility for analytics at attribute levels, but at the expense of greater indexing resources and storage for the increased number of flattened records. This is typically the better option for JSON files that do not have complex nesting.

Horizontal

Horizontal expansion is a flattening that typically benefits storage space by creating one record with many columns for each corresponding JSON record. This is typically the better option for JSON files that use nested arrays.

  • Schema Filter opens a window with features for virtually transforming the data type of a column, or for indexing (whitelisting) or excluding (blacklisting) JSON attributes from the indexed files. For a JSON file, the whitelist feature can be helpful to define special expansion rules for one or more attributes/columns in a JSON file when all the others use the opposite expansion. See Recommendations for AWS CloudTrail and Similar Records Arrays for an example.

👍

Some Recommendations for Object Groups

  • If you are indexing a JSON file that is relatively flat (that is, no complex arrays or nested arrays), vertical expansion typically offers the best index flattening and analytic balance.
  • If you are indexing a JSON file that has objects with nested arrays, horizontal expansion is usually the better option to take advantage of storage efficiency. You can use the ChaosSearch JSON Array Transformation to vertically expand specific columns of the index view for filtering or visualization needs.

Did this page help you?