The JSON Flex Advantage

JSON Flex® leverages the advantages of the Chaos Index® technology to uniquely represent the dimensionality of JSON by giving administrators the tools to selectively control how the JSON source files are indexed, creating the highly optimized Chaos Index data that drives query performance and information value for the end-user analysts.

JSON Flex tackles the JSON indexing and searching challenges from two important vectors:

• Flexible Chaos Index® choices that can process and efficiently store JSON file structures in the patented ChaosSearch indexed data format
• Flexible and query-performant Chaos Refinery® views with schema-on-read transformation features that empower analysts and end-users to specify, search, and visualize their important business data

JSON Flex allows customers to store raw JSON and analyze it as if it were structured at different nested levels — with options to manage and control the permutation explosion, no complex and unwieldy queries, and control over data granularity in analytics.

JSON Flex can help to maintain the smallest possible data representation at index time, while allowing users full customization at query time to materialize the data.

The Indexing Features

The JSON Flex solution begins with versatile configuration options that define how to index your JSON files. When you define an object group, you specify which JSON log and event files to index with that group, and you have the standard features like object filtering, isolation keys for data separation, and field overrides to control data typing of fields.

JSON FLEX augments object group processing features with these additional capabilities:

• Array flattening options—For each object group, you can specify:

  • A default group expansion choice of either horizontal or vertical expansion, with options to expand specific arrays differently
  • An array depth level to specify how deeply to flatten the properties contained in nested arrays at levels 1-10, all, or to flatten no arrays. Any arrays that are not flattened are indexed and stored as JSON strings, so their data is still captured within the Chaos Index content.

• Array selection controls—Within the JSON files, there will be arrays that contain data that is important for analysis, arrays that are not valuable for analysis, and options where you might want to have some arrays horizontally expanded and some vertically based on their content. JSON Flex includes array selection and control policies that you can use to include or exclude arrays by name and expressions.

  • Array selection policies—You can use selection rules to specify the arrays that you want to index (whitelist), or the arrays to skip (blacklist).
  • Vertical selection overrides—If you configure a horizontal expansion as the default for all arrays in an object group, you can override the setting for specific arrays to expand them vertically, and include their properties as filters and analysis fields.

• JSON nested object controls—JSON files could have nested objects that have nested properties. Nested objects are indexed as separate properties, which could result in a large number of separate fields in the indexed data. With JSON Flex, you have options to index nested properties as one JSON string, rather than as nexted individual properties.

The Refinery View Features

The JSON Flex solution continues with flexible processing options in Refinery views to give users and analysts options to control how the JSON indexed data can be used for search and visualizations. Views have many standard features for selecting the indexed data to include, to set timeframes for analysis windows, to filter on desired fields and values, and to configure different types of schema-on-read transformations that can morph indexed data fields into the key search columns and filters for the end users who analyze the data.

The JSON Flex view features include these additional capabilities for post-index processing:

• JSON array virtual transformation—When creating a view for an object group that uses JSON horizontal array expansion for storage efficiency, users might want to vertically flatten one or more of the horizontal arrays to make their properties available for the view transformations, filter controls, and displayable columns for queries against the views. Using the JSON Array Transformation window, you can select one or more horizontal arrays to virtually vertically transform.

• Schema transformations to materialize columns from the indexed fields—Within the Refinery view, you can configure transformations for JSON string indexed data.

  • You can use Materialize with JSONPath to create a materialized column with values extracted from a JSON string blob field. This transformation uses the JSON Path expression syntax to identify the JSON sting values.
  • You can use Treat as Nested JSON to specify that an index filed processed as one JSON string blob field can be used in a Search Analytics (OpenSearch Dashboards) filter or Elastic API search using an Elastic nested path expression.