Whether you use ChaosSearch Analytics and its Kibana offering to search and visualize data, or Elastic APIs to extract results for use with other applications, the Refinery® views help you to access the business and value insights in your JSON files. The controls for managing the flattening, storage, and filtering for JSON files gives your data analysts and BI consumers much more control over the storage and visualization.
This topic provides some examples of how Kibana visualizations might appear with horizontal and vertical expansion of a sample AWS CloudTrail log file.
In this example, the CloudTrail log is filtered to an object group that uses horizontal expansion. The Refinery view for the group does not use JSON Array Transformation to perform any virtual vertical transformations of the
For this configuration, the Kibana Discover report for the CloudTrail log data shows one record where all JSON properties are flattened to separate columns. The Discover output shows the first column sequence for
4 have the same structure.)
For a horizontal index, the possible visualizations are limited. You can use simple filters to search for specific values, but it is not possible to show aggregate analysis. For example, if you wanted to create a pie chart to show the number of different
eventName records in this log sample, the horizontal flattening does not support the ability to split the bucket by
eventName. The available
Terms to choose from are specific to each attribute column (
Records.0.eventName), as in the following example:
For horizontal expansion, the options for querying and visualization are basically string matches to find content within a record, or visualization of a single column value like
In this example, the CloudTrail log is filtered to an object group that uses vertical expansion. For this configuration, the Kibana Discover report shows 5 records in the CloudTrail log sample. The
Records array members are flattened to rows for each array member:
With vertical flattening, the granularity of the data allows for more analysis options. For example, to create a chart of the records and show a distribution by CloudTrail eventName, you can split the pie chart by the
The advantage of JSON Flex is that you have layers of flexibility in the options for the setup of your object groups and views. You can choose horizontal expansion for object groups to efficiently store the indexed data, then use a virtual vertical expansion on an attribute-by-attribute basis using the JSON Array Transformation window.
A sample window follows where the CloudTrail
Records attribute is configured for Vertical expansion in the view:
With this set of options, the discovered data for the log appears similar to the data for an index that was configured to use vertical expansion, but the index itself uses less storage.
For this example, the virtual vertically expanded
Records properties are selectable for graph filtering:
With the JSON array transformation feature, you can take advantage of better analysis controls for the JSON properties that users want the most.
If you create an object group and select an Array Flatten Depth value other than Unlimited, some of the JSON content might be indexed as an native JSON blob. As an example, using the same CloudTrail log file, if you create an object group of either expansion method and you specify an Array Flatten Depth of None, the resulting nested
Records object is flattened into a
Records column that contains the native JSON structure concatenated as a string, for example:
This type of native JSON content can be searched for string values, but the properties are not available for filtering options or other visualization capabilities.
Updated 11 days ago