Refinery Views

Create Refinery views to clean, prepare, and materialize the indexed fields associated with an object group into columns for analytics.

ChaosSearch Refinery views are logical indexes based on the physical indexes created in the Storage area of the ChaosSearch platform. Refinery views operate similarly to Kibana index patterns, or tables and views in SQL, but with additional ChaosSearch post-processing features for analytics.

You use the Refinery to select the indexed data to include in a view, and to prepare, filter, and materialize the indexed fields into columns for analytics. Schema transformations can be applied to one or many fields, allowing users to tailor the data for visualizations without the need to re-index any source data.

Working in the Refinery

Creating a View

The Refinery page includes a list of defined views on the left. Users can typically see only a permitted subset of the existing views.

To create a view:

  1. Click Create View in the top right of the window.

A create view window appears.

  1. Select one or more object groups in the left list. Your view can include the indexed data from one or more object groups. The right pane updates with the list of indexes in the selected group(s).
  2. You can narrow the view down to specific indexes within the selected object group(s). To filter the list of indexes, you can use the Index Pattern field to type a string that will display only the matching indexes.
  3. Optionally, select Index Window and specify a time range in days or months. The indexed data for the view will be limited to the specified index window range, and any index data files with older data will be removed from the list. See Index Window later in this topic for more information.
  4. After you have refined the list of indexes to include in the view, click Next.



The Index Window setting is disabled/not selected by default. If you select and set a window, the visualizations against this view include only the indexed data for that window even if the object groups have a longer history.

  1. In the Schema Transformation window, apply any transformations or filters needed for the indexed data. See Filtering Data in the View later in this topic for general filtering, or Schema Transformation for more information about materializing columns.
  1. Click Next.
  1. In the Timestamp Field, select the timestamp field to use for the view, which will be the timestamp used for histograms in Kibana.
  2. Click Create View. The following window appears:
  1. Type a unique name for the view. The view name will become available for analytics and search tasks. View names must comply with AWS naming conventions.
  2. Optionally, select any of the following options:
  • Cacheable to enable the query cache feature, which caches the results of queries using this view. When enabled, after an initial query is executed, results are cached for improved search results and experience for subsequent similar queries.
  • Overwrite to save your view with its current name even if a view with that name already exists. This allows you to replace an existing view of the same name with the new view and its settings.
  • Case insensitive specifies whether users must input the exact case when searching on terms in the data set. If enabled, searches will not require users to match exact letter case when searching, but queries could take longer to run because case-insensitive requires more processing.
  1. Click Create to add the new view to the Refinery.

View Summary

In the main Refinery window, select a view in the left list to display details about the indexes it references and the view configuration.

The View Summary tab on the right provides information about the Identifier (Index ID), whether a Time Field was set, and whether Case Insensitivity is set. These three fields are important to review especially if searches are not returning the expected results or a histogram is not present when working in the Analytics tools.

You can also edit the view to change the Index Window setting if needed.


Index Window

ChaosSearch and its cost-effective indexing allows you to keep a longer history of indexed data, but you might want to limit search results and analysis with this view to the last 14 days, for example, to balance query performance and to focus the user analysis to the most recent data.

The Index Window is a rolling time period that limits a view, and the corresponding analytics and visualization results, to the specified time window range. By default, Index Window is unselected/off, so queries that use the view could return results from across all of the indexed data that is available for the object group(s) on which the view is based.


If you enable the Index Window, the default value is 14 days. Also, the index list updates to show only the indexes that have data for that window range.

This window specifies that visualizations for the view could show data for any time inside the range of the last 14 days. If a user tries to view data for any time before the last 14 days, the view returns no results. Index windows help to focus users within a range that provides the analysis content that they typically need, and can help overall system performance.

You can specify a window when you create a view, and you can modify the window as needed to update for analysis window changes.

To modify the Index Window for an existing view, navigate to the Refinery and select the view from the left-hand menu. Click View Summary and modify the Index Window setting as needed.

Defining Case Insensitivity

When creating a view, you can select whether to enable case-insensitivity when executing searches on the new view. By selecting Case insensitive, users can search against terms without restrictions to the letter casing of the terms, but queries can require more time to run versus case-sensitive queries for the same filters.

Filtering Data in the View

When creating a view, you can specify filters for one or more columns to reduce the search results for the users querying that view. The view will not show any records that do not meet the filter criteria. For example, you might want to filter the results to show records only for specific applications, cloud regions, or similar values to focus the view results only on those criteria for the end users.


View Filters

The filters that you set when you create the view are a type of hard filter -- the view will always be limited to the matching records for any configured filters. For example, if you create a filter that limits results to a specific cloud region, users cannot see search results for any other cloud regions that are outside the filter.

These filters are not the same as the ad-hoc filters like in Kibana where users can specify a filter to narrow the search results in their browser. Users can set and change those ad-hoc filters as needed to tune results, but the results will never include records that do not meet the overall view filters.

To create a filter for a column:

  1. While in the create view process, and in the Schema Transformation window, select or search for the field on which you want to apply a filter.
  2. Click the filter icon at the right end of the field row. The Schema Transformation window opens.
  1. In the Configure Filter window, type a value to filter (or narrow) the results that are displayed by the view. This example filters a client_ip field to match a specific IP value. You can specify an exact value or use an asterisk (*) wildcard value for the filter value. The view results will include only the records that meet this filter criteria.

  2. Click Create to save the filter.

  1. The Schema Transformation window displays the filter you created below the field name.
  1. Optionally, you can repeat the steps to add more filters for the same field or for different fields.

  2. After you add another filter, the Schema Transformation page updates with the new filter value. The view treats multiple filter values for the same field as an "OR" condition; that is, records that match any of the filter values will be within the scope of the view results. Notice that each filter has a delete "x" icon that you can use to remove the filter if desired before you save the view.



You cannot change the filters after you create the view. You can, however, create a new view with different filter choices and overwrite/replace an existing view with the same name.


Schema Transformation

When creating a view, you can virtually transform any of the fields within the indexed data to create one or more materialized columns for analytics and analysis. These transformations are a powerful feature of the ChaosSearch Refinery views that enable post-processing of the indexed data fields within the context of a view.

To create a transformation:

  1. While in the create view process, and in the Schema Transformation window, select or search for the field to transform.
  2. Click the gear icon at the right end of the field row. The Schema Transformation window opens.

See Schema Transformations for more information about the available transformations within the Refinery views and how to use them.

Defining Case Insensitivity

After selecting the view, you will have the opportunity to select whether or not you want to enable case-insensitivity when executing searches on the new view. By selecting Case insensitive, users can search against terms regardless of case.


Case Insensitive searches

If the toggle is not selected searches will require you to input the exact letter case when searching on the data set. If the toggle is selected, searches will not require you to input exact case when searching.

Query Cache

The ChaosSearch query cache feature is used for caching the results of queries. If you enable caching, after an initial query is executed, results are cached for improved search results and experience when similar queries are re-run against the view.

Disable Query Cache

If an index does not require caching, you can simply create a view with cache disabled (not selected).

What’s Next