Refinery Views

Create Refinery views to select the data for visualizations and to virtually transform schema details for analytics.

ChaosSearch Refinery views are logical indexes based on the physical indexes created in the Storage area of the ChaosSearch platform. Refinery views are used instead of Kibana index patterns.

You use the Refinery to clean, prepare, and transform the index data. Schema transformations can be applied to one or many fields, allowing users to change data types to tune the data for visualizations without the need to re-index data.

Working in the Refinery

Creating a View

The Refinery page includes a list of defined views on the left. Users can typically see only a subset of the existing views.

To create a view:

  1. Click Create View in the top right of the window.

A create view window appears.

  1. Select one or more object groups in the left list. The right pane updates with the list of indexes in the selected group(s).
  2. To filter the list of indexes, you can use the Index Pattern field to type a string that will display only the matching indexes.
  3. Select Index Window and specify a time range in days or months to limit the view to visualizing the indexed data only within that specific window range. The index list updates to show only the indexes that have data for that range. See Index Window later in this topic for more information.
  4. After you have refined the list of indexes to include in the view, click Next.

🚧

NOTE:

The Index Window setting is disabled/not selected by default. If you select and set a window, the visualizations against this view can include only the indexed data within that window

  1. In the Schema Transformation window, apply any data file transformations or filters needed for the indexed data. See Schema Transformation later in this topic for more information.
  1. Click Next.
  1. In the Timestamp Field, select the timestamp field to use for the index.
  2. Click Create View. The following window appears:
  1. Type a unique name for the view. The view name will become available in the Kibana Analytics > Discover area. View names must comply with DNS naming conventions.
  2. Optionally, select any of the following options:
  • Cacheable to enable the query cache feature, which caches the results of queries using this view. When enabled, after an initial query is executed, results are cached for improved search results and experience for subsequent similar queries.
  • Overwrite to save your view with its current name even if a view with that name already exists. This allows you to replace an existing view of the same name with the new view and its settings.
  • Case insensitive specifies whether users must input the exact case when searching on terms in the data set. If enabled, searches will not require users to specify the exact letter case when searching.
  1. Click Create to add the new view to the Refinery.

View Summary

In the main Refinery window, select a view in the left list to display details about the indexes it references and the view configuration.

The View Summary tab on the right provides information about the Identifier (Index ID), whether a Time Field was set, and whether Case Insensitivity is set. These three fields are important to review especially if searches are not returning the expected results or a histogram is not present when working in the Analytics tools.

You can also edit the view to change the Index Window setting if needed.

Index Window

The Index Window is a rolling time period that limits a view, and the corresponding analytics and visualization results, to the specified time window range. By default, the Index Window is disabled/off, so queries that use the view could return results all of the indexed data that is available for the object group(s) on which the view is based.

If you enable the Index Window, the default value is 14 days. This window specifies that visualizations for the view can show data for any time inside the range of the last 14 days. If a user tries to view data for any time prior to the last 14 days, the view returns no results even if there is indexed data available. Index windows help to focus users within a range that provides the analysis content that they most typically need, and can help overall system performance.

You can specify a window when you create a view, and you can modify the window as needed to update for analysis window changes.

To modify the Index Window for an existing view, navigate to the Refinery and select the view from the left-hand menu. Click View Summary and modify the Index Window setting as needed.

Defining Case Insensitivity

When creating a view, you can select whether to enable case-insensitivity when executing searches on the new view. By selecting Case insensitive, users can search against terms without restrictions to the letter casing of the terms.

Schema Transformation

When creating a view, you can virtually transform the schema of any of the fields within the index structure. This enables you to create a more specific on-the-fly data typing to improve the analysis phase.

To create a transformation:

  1. While in the create view process, and in the Schema Transformation window, select or search for the field to transform.
  2. Click the gear icon at the right end of the field row.
  1. In the Schema Transformation pop-up, you can select from several predefined transformations. For an IP address, you can select Treated as IP, then click Save Transform.

Other options include:

  • Materialize with Regex to use a custom regular expression to transform a field, perhaps into several searchable fields with distinct content.
  • Treat as GeoPoint for geo-location data.
  • Treated as Partition Key to set the field as a partition key for the index.

As an example of a regular expression transformation, if you have a log file with a column of web URL data, you could transform the field into three virtual fields of domain, port, and path for analytics.

After selecting the URL field and clicking the gear icon to transform it:

  1. Make sure that Materialize with Regex is selected as the transformation.
  2. Type the regular expression pattern to use, such as (\S+[ :])(\d+)(\S+) as an example. See Refinery Transformation Regex for other sample patterns.
  3. Click Field to add three fields, and name them domain, port, and path. Make sure that domain and path are STRING types, while port is a NUMBER type.
  4. Click Refresh to update the Preview pane and review the transformation. If there are any errors or changes, you can update the regular expression and/or fields and refresh again.
  5. Click Save Transform when finished.

After saving your transformation changes, the Schema Transformation window shows the transformed fields as in the following example:

Defining Case Insensitivity

After selecting the View, you will have the opportunity to select whether or not you want to enable case-insensitivity when executing searches on the new view. By selecting Case insensitive, users can search against terms regardless of case.

📘

Case Insensitive searches

If the toggle is not selected searches will require you to input the exact letter case when searching on the data set. If the toggle is selected, searches will not require you to input exact case when searching.

Query Cache

The ChaosSearch query cache feature is used for caching the results of queries. After an initial query is executed, results are cached for improved search results and experience when similar queries are re-run against the view.

Disable Query Cache

If an index does not require caching, you can simply create a view with cache disabled (not selected).


Did this page help you?