JSON Flex Behavior Summary

JSON Flex offers a combination of pre- and post-indexing settings and features that can help to efficiently index even complex JSON log files—all without re-pipelining the raw JSON files. The options offer a combination of advantages and benefits to the users.

As a general best practice, it can be very helpful to test some sample JSON files and try out the JSON Flex options to see which expansion and flex options might yield the best results for your JSON source files.

Start with some representative JSON data (perhaps a few hours of JSON data) and use the ChaosSearch indexing features to create object groups with different JSON Flex options so that you can assess storage choices and analysis differences. Similarly, you can create some test views for your sample object groups to review the indexed data results, to see the types of filters and columns that users will be able to access and use, and to try out the various JSON transformations that might create just the right level of filtering and visualization in the analytics phase.

Object Group Setting	Refinery View Setting	Pro's	Con's
Vertical Expansion		All JSON properties can be used as filters for analysis and reporting.	Usually the largest storage requirements for the indexed data for complex nested arrays. Some JSON fields might not be valuable for analysis filtering; Some duplication of rows during array flattening could impact aggregations and counts.
Horizontal Expansion	In a view, JSON Array Transformation can be used to select horizontally flattened arrays to virtually vertically expand them to make their properties available as search filters.	Horizontal expansion is usually the best option for the smallest indexed data storage footprint for complex nested arrays.	Horizontally expanded array elements and properties are not well suited for filtering or aggregate analysis due to the separated columns for array members.
Horizontal with vertical expansion whitelist. Hybrid of horizontal expansion; by default all arrays are horizontally expanded during indexing except the arrays referenced in the `vertical_selection_policy`.	Can use JSON Array Transformation for the horizontal arrays found in the indexed data. Selected vertical expansions offer the analysis gains to make their properties available as filters.	Horizontal expansion yields the index storage gain, but a vertical storage can sometimes be more efficient for very complex arrays that would otherwise flatten horizontally to millions of columns.	Horizontally flattened array attributes are not well suited for filtering or aggregate analysis. Increased storage needs could result for the vertical expanded whitelist attributes.
Horizontal or vertical expansion with Array Flatten Depth other than Unlimited.	No expansion for any nested arrays at levels below the specified depth level. An unexpanded array and its properties are stored as a JSON string value in one column. None specifies that all arrays are indexed as JSON string values.	Helps reduce row and/or column counts for a record when text searches within the JSON string content is sufficient for analysis.	JSON string values cannot support filtering or visualization operations, unless Materialize with JSONPath or Materialize with JQ is used. JSON string fields can support searches, including searches with Elastic nested query syntax.
Field selection policy can be used to specify nested JSON objects (not arrays) to be indexed and stored as JSON strings.		Helps reduce the conversion of nested object properties to many columns or filtering fields that might not be very useful.	JSON string values cannot support filtering or visualization operations, unless Materialize with JSONPath or Materialize with JQ is used. JSON string fields can support searches, including searches with Elastic nested query syntax.
	Materialize with JSONPath or Materialize with JQ schema transformation can be used to materialize a JSON field stored inside a JSON string to create a virtual column in the view for analytics and filtering.	For important content inside the JSON string, you can make that property into a virtual search filter and materialized column.
	Treat as Nested JSON schema transformation allows a JSON string column to support searches with Elastic nested query path syntax.	For important content inside a JSON string, you can enable that field to support nested query path expressions to search for values with Discover or the Elastic API.