The Discover page searches are usually the first step for new ChaosSearch users, or those who are searching for specific data in their log and event files.

There are three basic search types in Kibana:

• Free text searches – run a wide search for important values
• Field-level searches – search specific fields for values of interest
• Filter-based searches – search fields with UI filtering controls that can help to narrow the results

The Kibana/Elastic applications also include support for wildcards, ranges, and regular expression matching options. This topic describes the options and their uses for creating more granular search results.

Free Text Searches

A free text search scans all the indexed data for a specific Refinery view, across all columns, for a specific time frame. Free text searches are very wide, and can be helpful if you are generally looking for values of interest, but you are not sure about the fields where those values might be stored.

Depending on your text string, free text searches could return a very large number of results. The results all have some relationship to the search string, but it could be a lot of work for you to review those results for the relevant ones that you wanted, especially if you use wildcards to create a wider sweep of values. Searching for a term like error in large log file indexes over a long period of time could return millions or even billions of results, and require a lot of compute/worker resources to complete.

A free text search is used as the basic search example in Step 4. Search and Visualize your Indexed Data. Free text searches can be useful to run a general sweep of the indexed data for special values of interest. When you want to narrow down the results of your search, you can add field-level and filter searches.

👍

Hits, Results, Documents...

Hits, results, records, rows, documents, and data are synonyms for the information returned in a search of the indexed data associated with a Refinery view. Elastic uses the term document (or doc) for the content that it searches and a matching result returned in Elastic queries. Kibana uses hits as the number of items found in a Kibana discover/search of content. SQL uses rows (and columns) for the information returned in SQL SELECT operations, though the term record is sometimes used. All of these terms are used in the ChaosSearch documentation to be flexible with the supported UIs, APIs, and possible integrations with third-party apps that might use these or other terms for results.

Field-Level Searches

In a field level search, you query for values in specific fields. If you know which field contains the information that you are searching for, field level searches can return a more granular set of results, and in less time than a free text search across all columns.

For example, if you want to find which orders records were processed by Clerk#000000497, type o_clerk:Clerk#000000497 in the search window and refresh. For this sample data, five results are returned.

You can combine the field-level search criteria with AND, OR, and NOT syntax to create even more granular searches. For example, to search for the records from a specific clerk that had a 5-LOW o_orderpriority, you could run a search similar to the following.

Only one record has that combination of clerk and priority values. With a good use of field searches, you can reduce the results to a much more granular set for review.

The NOT syntax is a helpful way to filter out conditions. For example, to search for records of orders that were above a certain total price and associated with all clerks except one specific ID:

o_totalprice > 200000 AND NOT (o_clerk:Clerk#000000250)

Filter Searches

Filter-based searches are similar to the field-based searches, but they offer a GUI and some helpful options and controls to enable or disable them for search refinement. Filters are additive, meaning that they work in combination with each other, and also with any free text or field-level search criteria you specify. The results must match all of the text and filter criteria to qualify.

For example, you might want to know how many of the orders processed by a clerk have an order priority of 1-URGENT. You can add a filter of o_orderpriority:1-URGENT by clicking the field name under Available fields, which displays a pop-up of the top values for the present search. Click the plus sign next to 1-URGENT to select that value for the query.

📘

Filter options

The options in the filter pop-up are based on the values available in the currently displayed results. If a desired filter option is not in the pop-up, expand the search results to see more filter values in the pop-up, or type the desired filter as a field-level search value.

The filter is added to the Add filter list at the top of the window, and the query updates to display the new search results for the specified clerk ID and priority.

In this search test, there is only one result. If you want to remove the priority filter and return to the prior set of results, click the x after the o_orderpriority filter at the top to remove the filter and refresh the query.

Using the field search method, you could specify multiple search filter options by and'ing the conditions. You could use OR to filter for results by matching on one of the OR options. To ensure that the AND and OR strings are processed correctly, use parentheses when needed to keep the comparisons together. For example, to list results for a specific clerk that are either of urgent or medium priority:

o_clerk:Clerk#000000497 AND (o_orderpriority:1-URGENT OR o_orderpriority:3-MEDIUM)

The free text, filter, and field options can be used in any combination to refine your search queries. As you use the search options more, you can start to develop useful combinations and save your favorite queries. By using saved queries, you can load your favorite or most used search combinations, either to quickly run those searches or to edit the saved queries to adjust the filters or time range. Saved searches are also helpful for creating visualizations of your favorite queries.

Search Options

Kibana and Elastic support some logic options for the search term values. The following table describes different options for search terms that provide some flexibility in values.

Note that KQL does not support regular expressions or searching with fuzzy or proximity terms. ChaosSearch does support some regular expressions in Lucene searches.

Histogram Display and Time Settings

At the top of the search results page is a histogram of the results over the query time range. The histogram is displayed when the Refinery view has a defined timestamp field (selected when the view was created). Some files might not have timestamps—those files are typically dimension tables with fields that contain supporting information. Queries against views that do not have a timestamp will not display the histogram, nor the time range selector (since it is meaningless for those tables). As an example, region-view contains information with no timestamp details – a dimension table. If you query that view, the histogram and time ranges do not appear.

Setting Time Ranges

The default time range for a search is the last 15 minutes. It is very easy to change and adjust the time ranges using the Kibana controls to select other ranges that better suit the index data or the type of investigation that you are performing.

For example, you can click the calendar icon next to the time range value to see a pop-up window with some easy-to-select options including an adjustment for the last number of minutes, or for selecting from commonly used values, or by selecting from the range values used in recent searches:. There is also a Refresh option to update the data on a scheduled time frame, which can be helpful for live object groups and the continuously new data that they receive. (Refresh is not very useful for searches on static indexes, because as the name implies, that data typically is not changing over time.)

You can click the starting or ending time range to display another set of time options to set the start or end time value using a calendar widget or a relative interval widget, or to set a time to "now" using a quick click.

Alternatively, the histogram shows a window of data over the current time period. Click and drag inside of the histogram to zoom to a more specific duration of time. The time picker updates for the selected range, and the histogram updates to show the new data points.

Query Your Data

The power of ChaosSearch is the ability to effectively transform S3 objects into searchable entities. You can type Kibana Query Language (KQL) into the search bar above the histogram to refine the results.

Enter your desired search criteria into the search bar and press Enter.

👍

Lucene Query Syntax

Elasticsearch uses the Lucene query syntax for searches. Refer to query string syntax for more information about constructing your queries.

Explore File Details

Click the expand button (carat) on the left of any row in the search results to view all file data as a table.

By default, the data is displayed in a tabular format. To view the data as a JSON object, click the JSON tab.

Save a Query

Saving queries allows you to name and quickly reload the query and any filters in the Discover page. You can use the saved query to quickly load a favorite visualization.

To save a query:

1. Click the disk icon. The Saved Queries window opens.

2. Click Save current query. The Save query window opens.

3. Type a name for the query. This example uses AppCost as the name. You can also choose to add a description string, and whether to save the filters and the time range used for the query that you are saving.
4. Click Save.

The query is added to the list of queries that appears when you click the carat next to the disk icon.

You can delete a saved query by clicking the disk icon to open the list, and then clicking the trashcan icon to delete the saved query. You will be prompted to confirm the deletion.

Filtering the Record Display

You can use the Available fields selections in the left list to reduce and tune the data that appears for the records returned below the histogram. For example, you could select o_clerk and o_orderpriorty to show only those two columns in addition to the timestamp (when time is part of the view):

The filter fields that you select can be cleared by hovering over each one and clicking the x to remove that option from the Selected fields area on the left. If you remove all of the selected options, the record display returns to its complete form.

👍

More information

See the Kibana documentation for more information about exploring your data using Discover.