Whether you use ChaosSearch Analytics and its Kibana offering to search and visualize data, or Elastic APIs to extract results for use with other applications, the Refinery® views help you to access the business and value insights in your JSON files. The controls for managing the flattening, storage, and filtering for JSON files gives your data analysts and BI consumers a high degree of control over the storage and visualization options.
This topic provides some examples of how Kibana visualizations might appear with horizontal and vertical expansion of a sample AWS CloudTrail log file.
In this example, the CloudTrail log is filtered to an object group that uses horizontal expansion. The Refinery view for the group does not use JSON Array Transformation to perform any virtual vertical transformations of the
For this configuration, the Kibana Discover report for the CloudTrail log data shows one record where all JSON properties are flattened to separate columns. The Discover output shows the first column sequence for
4 have the same structure.)
For a horizontal index, the possible visualizations are limited. You can use simple filters to search for specific values, but it is not possible to show aggregate analysis. For example, if you wanted to create a pie chart to show the number of different
eventName records in this log sample, the horizontal flattening does not support the ability to split the bucket by
eventName. The available
Terms to choose from are specific to each attribute column (
Records.0.eventName), as in the following example:
For horizontal expansion, the options for querying and visualization are basically string matches to find content within a record, or visualization of a single column value like
In this example, the CloudTrail log is filtered to an object group that uses vertical expansion. For this configuration, the Kibana Discover report shows 5 records in the CloudTrail log sample. The
Records array members are flattened to rows for each array member:
With vertical flattening, the granularity of the data allows for more analysis options. For example, to create a chart of the records and show a distribution by CloudTrail eventName, you can split the pie chart by the
The advantage of JSON Flex is that you have layers of flexibility in the options for the setup of your object groups and views. You can choose horizontal expansion for object groups to efficiently store the indexed data. Within the view definition, you can choose a virtual vertical expansion for fields to materialize the columns for the view using the JSON Array Transformation window.
A sample window follows where the CloudTrail
Records attribute is configured for a virtualVertical expansion in the view definition:
With this set of options, the materialized
Record columns for the view are similar to how they would be if they had been vertically expanded in the object group's index fields. The
Record columns can be used for filtering and aggregations, even though the indexed data is stored in its horizontal expansion.
For this example, the virtual vertically expanded
Records properties are selectable for graph filtering:
With the JSON array transformation feature, you can take advantage of more granular JSON properties for filtering to improve the usability for user analytics.
If you create an object group and select an Array Flatten Depth value other than Unlimited, some of the JSON array content will be indexed as a native JSON string value. As an example, using the same CloudTrail log file, if you create an object group of either expansion method and you specify an Array Flatten Depth of None, the resulting nested
Records object is flattened into a
Records column that contains all the native JSON structure concatenated as a string, for example:
This type of native JSON content can be searched for string values. Normally, the JSON properties in the string would not be accessible as filter or aggregation columns. However, using the ChaosSearch Materialize with JSON view transformation option, you can define JSON properties within the string to materialize as columns in the view. See Schema Transformations for more information about this feature.
Updated 18 days ago