Walkthrough: Creating Views with JSON Flex Features

ChaosSearch Refinery® views have many features for selecting, transforming, and materializing the business insights data captured within the Chaos Index® data.

This topic is a high-level walkthrough of the UIs and steps to create views for querying, searching, and visualizing the data indexed from your JSON cloud-storage files. Read about the view workflow and the UIs that provide the JSON Flex capabilities. Details about the UIs and concepts are available in linked topics below.

Select the Indexed Data for the View

When you create a view, the first step is to select the indexed data to include in the view. You can select one or more object groups and then select one or more daily intervals. The intervals are the daily statistics and related ChaosSearch indexed data for the log and event files that were indexed by the selected object group(s).

After you choose the intervals for your view, the interface walks you through the options available for that data.

JSON Array Transformation

If the intervals that you select have any horizontally expanded arrays, the JSON Array Transformation window appears. You can select an array that was expanded and stored horizontally in the index data, and virtually transform the array to a vertical expansion when it is used in the view.

The arrays that you select for transformation virtually expand to become available as view filtering values, as columns for visualizations, and as options for some types of aggregations like sum and count aggregations. You have the benefit of the horizontal index storage, and the advantage of the vertical visualization capabilities.

📘

No horizontally expanded arrays? No JSON Array Transformation window.

The JSON Array Transformation window does not appear unless there is at least one horizontally expanded array. The view creation workflow skips the window display if it is not applicable.

Filters and JSON Flex Transformations

After the JSON Array Transformation window (if is was applicable for the view), the view workflow takes you to the Schema Transformation window. The window lists all of the columns that the view will support for searching, filtering, column display, and visualizations.

In the example, note that this sample AWS CloudTrail data shows columns that all start with aRecords. prefix. CloudTrail logs typically have a top-level Records array and all the usage and details are defined in fields of that array. ChaosSearch offers the ability to rename columns to make them shorter or easier to understand for users.

For each column of the view, you can use the Filter icon to specify one or more values to restrict in the search results for the view. The filters will display only the records that match the filter criteria and any other search criteria of your queries. For example, you could define a filter to limit the view and its search results to only those results that have a specific event source string value in the Records.eventSource column.

The gear/Transform icon opens a window where you can select transformations for powerful schema-on-write changes and materializations of the view columns. This is where you can take advantage of the JSON Flex Materialize with JSONPath, Materialize with JQ, and Treat as Nested JSON transforms.

These JSON Flex transformations help you to extract value even from complex JSON string blob fields; you can search for important analysis fields and values from inside the JSON string, or turn the string content into materialized columns of the view.

These JSON Flex options give you the full advantage of lossless array content storage in the indexed data, with compact string forms that reduce the indexed data storage size, and also reduce the impact of arrays on the possible JSON permutation challenge.

JSON Flex Materialize with JSONPath

You use the Materialize with JSONPath transform to define a materialized column that contains data extracted from a field inside a JSON string. Click Add Column to define a new column for the view and the JSONPath expression to extract the field from the JSON string. There are online tools and references with more information about how to build and test a JSONPath expression like the sample shown below.

After you specify a JSON Path value, click Refresh to update the Preview area and see if the JSON Path expression is finding the right data in the JSON string. With ChaosSearch, you can test the path before you save the view, easing the process to create the correct path expression.

Your new columns will appear in the view inventory and be available for users in Search Analytics and SQL Analytics.

JSON Flex Materialize with JQ

If an indexed data field contains JSON string content, you can use the jq transformation as an alternative to materializing with a JSON Path to specify one or more JSON properties as materialized columns for analytics. Materialize with JQ filtering works identically to the JSON Path materialization, but uses jq filters as the means to extract the field values for the new columns. The JSON string blob remains intact for text searches and other analysis.

After you specify a JQ filter expression value, click Refresh to update the Preview area and see if the jq expression is finding the right data in the JSON string. With ChaosSearch, you can test the path before you save the view, easing the process to create the correct path expression.

Your new columns will appear in the view inventory and be available for users in Search Analytics and SQL Analytics.

JSON Flex Treat as Nested JSON

If view columns are not required, but administrators want deeper search controls over the JSON string and properties, you can transform the string column into one that can be used with the Elastic nested query path searches.

The Treat as Nested JSON transform configures a JSON string column to support Elastic nested path expressions to query for values as a Search Analytics > Discover filter or in Elastic API search calls.

This transform might be very helpful for users who are used to running Elastic nested queries; they can now perform the same kinds of search operations on their JSON log and event data.