The JSON Flex Advantage

JavaScript Object Notation (JSON) is a detail-rich format for describing data. Many applications and platforms use JSON structured information to capture their user activities, application behaviors, and operations, creating a wealth of valuable business insights within an organization.

JSON Flex® leverages the advantages of the Chaos Index® technology to uniquely represent the dimensionality of JSON. ChaosSearch gives administrators the tools to selectively control how JSON source files are indexed without rounds of ETL toil, and provides users with flexible options to drive query performance and analysis.

JSON Flex tackles the JSON indexing and searching challenges from two important vectors:

• Flexible Chaos Index® choices that can process and efficiently store JSON file structures in the patented ChaosSearch indexed data format
• Flexible and query-performant Chaos Refinery® views with transformation features that empower analysts and end-users to specify, search, and visualize their important business data

JSON Flex allows customers to store raw JSON compactly, and analyze it as if it were structured at different nested levels — with options to manage and control the permutation explosion, no complex and unwieldy queries, and control over data granularity in analytics.

JSON Flex can help to maintain the smallest possible data representation at index time, while allowing users full customization at query time to materialize the data.

The Indexing Features

The JSON Flex solution begins with versatile configuration options that define how to index your JSON files. When you define an object group, you specify which JSON log and event files to index with that group, and you have the standard features like cloud storage object filtering, isolation keys for data separation, and schema overrides to control data typing of fields.

JSON FLEX augments object group processing features with these additional capabilities:

• Array flattening options—For each object group, you can control the array permutation explosion with:

  • A default group expansion choice of either horizontal or vertical expansion, with options to expand specific arrays differently
  • An array depth level to specify how deeply into nested arrays to flatten, with values from NONE, 1-10, or UNLIMITED depth. Any arrays that are not flattened are indexed and stored as JSON string blobs, so their data is still available within the Chaos Index content.

• Array selection controls—Within the JSON files, there will be arrays that contain important data for analysis, and arrays that have less meaningful information. You can choose to flatten the important ones, or choose to have some arrays horizontally expanded and some vertically based on their content. JSON Flex includes array selection and control policies that you can use to include or exclude arrays by name and expressions.

  • Array selection policies—You can use selection rules to specify the arrays that you want to index (whitelist), or the arrays to skip (blacklist).
  • Vertical selection overrides—If you configure a horizontal expansion as the default for an object group, you can use this policy to override the setting for specific arrays to expand them vertically, and include their properties as filters and analysis fields.

• JSON nested object controls—JSON files could have nested objects with deeply nested properties. Nested objects are indexed as separate properties, which could result in a large number of fields in the indexed data. With JSON Flex, you have options to index nested properties in one JSON string column for compactness.

The Refinery View Features

The JSON Flex solution continues with flexible processing options in Refinery views to give users and analysts options to control how the JSON indexed data can be used for search and visualizations. Views have many standard features for selecting the indexed data to include, setting timeframes for analysis windows, filtering on desired fields and values, and configuring different types of schema-on-write transformations that can materialize indexed data fields into the search columns and filters for the end users who analyze the data.

The JSON Flex view features include these additional capabilities for post-index processing:

• JSON array virtual transformation—When an object group uses JSON horizontal array expansion for storage efficiency, users might want to vertically flatten one or more of the horizontal arrays to make those properties available for the view transformations, filter controls, and displayable columns for queries against the views. Using the JSON Array Transformation window during view definition, you can select one or more horizontal arrays to virtually vertically transform using that view.

• Schema transformations to materialize columns from the indexed fields—Within the Refinery view, you can configure transformations for JSON string indexed data.

  • You can use Materialize with JSONPath to create a materialized column with values extracted from a JSON string blob field. This transformation uses the JSON Path expression syntax to identify the JSON sting values that will become columns in the view.
  • You can use Materialize with JQ to create a materialized column with values extracted from a JSON string blob field. This transformation uses the JQ filter expression syntax to identify the JSON sting values that will become columns in the view.
  • You can use Treat as Nested JSON to specify that an index field processed as one JSON string blob field can be used in a Search Analytics (OpenSearch Dashboards) filter or Elastic API search using an Elastic nested path expression. Treat as Nested JSON fields can also be used in Search Analytics visualizations for Metrics or Bucket values, using a JSON Path value to identify the specific field in the blob.