Searching JSON Views

Some examples of Kibana searches and visualizations using horizontally and vertically expanded JSON log files

Whether you use ChaosSearch Analytics and its Kibana offering to search and visualize data, or Elastic APIs to extract results for use with other applications, the Refinery® views help you to access the business and value insights in your JSON files. The controls for managing the flattening, storage, and filtering for JSON files gives your data analysts and BI consumers much more control over the storage and visualization.

This topic provides some examples of how Kibana visualizations might appear with horizontal and vertical expansion of a sample AWS CloudTrail log file.

Example 1: Horizontally Expanded CloudTrail Log

In this example, the CloudTrail log is filtered to an object group that uses horizontal expansion. The Refinery view for the group does not use JSON Array Transformation to perform any virtual vertical transformations of the Records column.

For this configuration, the Kibana Discover report for the CloudTrail log data shows one record where all JSON properties are flattened to separate columns. The Discover output shows the first column sequence for Records.0.* (Records.1 through 4 have the same structure.)

For a horizontal index, the possible visualizations are limited. You can use simple filters to search for specific values, but it is not possible to show aggregate analysis. For example, if you wanted to create a pie chart to show the number of different eventName records in this log sample, the horizontal flattening does not support the ability to split the bucket by eventName. The available Terms to choose from are specific to each attribute column (Records.0.eventName), as in the following example:

For horizontal expansion, the options for querying and visualization are basically string matches to find content within a record, or visualization of a single column value like Record.0.eventName.

Example 2: Vertically Flattened CloudTrail Log

In this example, the CloudTrail log is filtered to an object group that uses vertical expansion. For this configuration, the Kibana Discover report shows 5 records in the CloudTrail log sample. The Records array members are flattened to rows for each array member:

With vertical flattening, the granularity of the data allows for more analysis options. For example, to create a chart of the records and show a distribution by CloudTrail eventName, you can split the pie chart by the Records.eventName attribute:

Example 3: JSON Array Transformations for Vertical Analysis and Horizontal Storage Benefits

The advantage of JSON Flex is that you have layers of flexibility in the options for the setup of your object groups and views. You can choose horizontal expansion for object groups to efficiently store the indexed data, then use a virtual vertical expansion on an attribute-by-attribute basis using the JSON Array Transformation window.

A sample window follows where the CloudTrail Records attribute is configured for Vertical expansion in the view:

With this set of options, the discovered data for the log appears similar to the data for an index that was configured to use vertical expansion, but the index itself uses less storage.

For this example, the virtual vertically expanded Records properties are selectable for graph filtering:

With the JSON array transformation feature, you can take advantage of better analysis controls for the JSON properties that users want the most.

Example 4: Array Flattening Depth and JSON Blob Values

If you create an object group and select an Array Flatten Depth value other than Unlimited, some of the JSON content might be indexed as an native JSON blob. As an example, using the same CloudTrail log file, if you create an object group of either expansion method and you specify an Array Flatten Depth of None, the resulting nested Records object is flattened into a Records column that contains the native JSON structure concatenated as a string, for example:

This type of native JSON content can be searched for string values, but the properties are not available for filtering options or other visualization capabilities.


Did this page help you?