Summary of JSON Flex Options

Review a matrix of the expansion and flatten depth options.

The JSON logs created by the cloud and orchestration services are typically very unique. ChaosSearch offers a combination of settings and features that can help to efficiently index even complex JSON log files. It can be helpful to test the JSON files and ChaosSearch indexing options to determine which options yield the best results for your files and analytics users.

Start with a subset of representative JSON data (perhaps a few hours of JSON data) and use the ChaosSearch Storage and Refinery features to create object groups with different flattening options so that you can assess storage choices and differences. Similarly, you can create test views to try different JSON transformations to find the right level of filtering and visualization in the analytics phase.

Object Group Setting

Refinery View Setting

Pro's

Con's

Vertical Expansion

All JSON properties can be used as filters for analysis and reporting.

Largest storage requirements for indexed data. Some properties might be less valuable for filtering and analytics.

Horizontal Expansion

JSON Array Transformation can be used to virtually vertically expand one or more nested arrays.

Horizontal array properties are best suited to text search types of analysis.

Vertically transformed fields support full filtering and analytics features.

Generally yields lowest indexed data storage requirements for the more complex nested JSON arrays.

Horizontal nested array properties are not well suited for filtering or aggregate analysis.

Horizontal with vertical expansion whitelist.

Hybrid of horizontal expansion for all columns except those referenced in the vertical expansion schema filter.

Can still use JSON Array Transformation for other horizontal arrays found in the indexed data.

Selected vertical expansions offer the analysis gains.

Horizontal attributes offer a storage gain, but a vertical storage can sometimes be more efficient for very complex arrays that flatten horizontally to millions of columns.

Horizontal nested attributes are not wesuited for aggregate analysis.

Increased storage needs for the vertical expanded whitelist attributes.

Horizontal or vertical expansion with Array Flatten Depth other than Unlimited.

No expansion for any nested arrays at levels below the specified depth level. Each array and its properties are stored as a JSON string value in one column.

None specifies that all array levels are indexed as native JSON strings.

Helps reduce row and/or column counts for a record when text searches within the JSON string content is sufficient for users.

JSON string values cannot support filtering or visualization operations (unless Materialize with JSON is used).

Field selection policy can be used to specify nested JSON objects (not arrays) to be indexed and stored as JSON strings.

Helps reduce row and/or column counts for nested object properties when text searches within the JSON string content is sufficient for users.

JSON string values cannot support filtering or visualization operations (unless Materialize with JSON is used).

Materialize with JSON schema transformation can be used to materialize a JSON property stored inside a JSON string to create a virtual column in the view for analytics and filtering.

For content that is stored as a JSON string, you can make that property into a virtual column.


Did this page help you?