Searching the Indexed Data

Many users begin by using the Discover page to search Refinery views for information.

The Discover page searches are usually the first step for new ChaosSearch users, or those who are searching for specific data in their log and event files.

There are three basic search types in Kibana:

  • Free text searches – run a wide search for important values
  • Field-level searches – search specific fields for values of interest
  • Filter-based searches – search fields with UI filtering controls that can help to narrow the results

The Kibana/Elastic applications also include support for wildcards, ranges, and regular expression matching options. This topic describes the options and their uses for creating more granular search results.

Free Text Searches

A free text search scans all the indexed data for a specific Refinery view, across all columns, for a specific time frame. Free text searches are very wide, and can be helpful if you are generally looking for values of interest, but you are not sure about the fields where those values might be stored.

Depending on your text string, free text searches could return a very large number of results. The results all have some relationship to the search string, but it could be a lot of work for you to review those results for the relevant ones that you wanted, especially if you use wildcards to create a wider sweep of values. Searching for a term like error in large log file indexes over a long period of time could return millions or even billions of results, and require a lot of compute/worker resources to complete.

A free text search is used as the basic search example in Step 4. Search and Visualize your Indexed Data. Free text searches can be useful to run a general sweep of the indexed data for special values of interest. When you want to narrow down the results of your search, you can add field-level and filter searches.

👍

Hits, Results, Documents...

Hits, results, records, rows, documents, and data are synonyms for the information returned in a search of the indexed data associated with a Refinery view. Elastic uses the term document (or doc) for the content that it searches and a matching result returned in Elastic queries. Kibana uses hits as the number of items found in a Kibana discover/search of content. SQL uses rows (and columns) for the information returned in SQL SELECT operations, though the term record is sometimes used. All of these terms are used in the ChaosSearch documentation to be flexible with the supported UIs, APIs, and possible integrations with third-party apps that might use these or other terms for results.

Field-Level Searches

In a field level search, you query for values in specific fields. If you know which field contains the information that you are searching for, field level searches can return a more granular set of results, and in less time than a free text search across all columns.

For example, if you want to find which orders records were processed by Clerk#000000497, type o_clerk:Clerk#000000497 in the search window and refresh. For this sample data, five results are returned.

14081408

You can combine the field-level search criteria with AND, OR, and NOT syntax to create even more granular searches. For example, to search for the records from a specific clerk that had a 5-LOW o_orderpriority, you could run a search similar to the following.

13751375

Only one record has that combination of clerk and priority values. With a good use of field searches, you can reduce the results to a much more granular set for review.

The NOT syntax is a helpful way to filter out conditions. For example, to search for records of orders that were above a certain total price and associated with all clerks except one specific ID:

o_totalprice > 200000 AND NOT (o_clerk:Clerk#000000250)

Filter Searches

Filter-based searches are similar to the field-based searches, but they offer a GUI and some helpful options and controls to enable or disable them for search refinement. Filters are additive, meaning that they work in combination with each other, and also with any free text or field-level search criteria you specify. The results must match all of the text and filter criteria to qualify.

For example, you might want to know how many of the orders processed by a clerk have an order priority of 1-URGENT. You can add a filter of o_orderpriority:1-URGENT by clicking the field name under Available fields, which displays a pop-up of the top values for the present search. Click the plus sign next to 1-URGENT to select that value for the query.

📘

Filter options

The options in the filter pop-up are based on the values available in the currently displayed results. If a desired filter option is not in the pop-up, expand the search results to see more filter values in the pop-up, or type the desired filter as a field-level search value.

13751375

The filter is added to the Add filter list at the top of the window, and the query updates to display the new search results for the specified clerk ID and priority.

13751375

In this search test, there is only one result. If you want to remove the priority filter and return to the prior set of results, click the x after the o_orderpriority filter at the top to remove the filter and refresh the query.

Using the field search method, you could specify multiple search filter options by and'ing the conditions. You could use OR to filter for results by matching on one of the OR options. To ensure that the AND and OR strings are processed correctly, use parentheses when needed to keep the comparisons together. For example, to list results for a specific clerk that are either of urgent or medium priority:

o_clerk:Clerk#000000497 AND (o_orderpriority:1-URGENT OR o_orderpriority:3-MEDIUM)

14301430

The free text, filter, and field options can be used in any combination to refine your search queries. As you use the search options more, you can start to develop useful combinations and save your favorite queries. By using saved queries, you can load your favorite or most used search combinations, either to quickly run those searches or to edit the saved queries to adjust the filters or time range. Saved searches are also helpful for creating visualizations of your favorite queries.

Search Options

Kibana and Elastic support some logic options for the search term values. The following table describes different options for search terms that provide some flexibility in values.

Note that KQL does not support regular expressions or searching with fuzzy or proximity terms. ChaosSearch does support some regular expressions in Lucene searches.

Search Option

Description

Wildcard (*)

The asterisk wildcard character replaces (represents) multiple characters. For example:

  • *reg* finds instances of the chars reg as is or within strings like irregular or regular and so forth.
  • reg* matches when reg are the first three characters are a field.
  • *reg matches when reg are the last three characters of a field.

By default, Kibana does not support leading wildcards for performance reasons, but ChaosSearch supports them.

Wildcard (?)

The question mark wildcard character replaces (represents) one valid character, number, or symbol. For example:

  • "*si?e*" matches on side, aside, sitewide, and others where ? could be any one value and the matching item could be any length.
  • "*?ide" matches on values that end a string in ide with any (or no) preceding characters.
  • "side?*" matches on values that begin a string with side and any number of characters like sides, siding, sitewide, or sideways.
  • "si?e *" (with a space before *) matches on strings that begin with the four-character value si-e, where the third character could vary.
  • "? *" will match results if a string begins with any one-character value.

When using the single-character wildcard, enclose the wildcard string in double quotation marks.

Ranges

In a KQL search, you can specify ranges using the range operators >, >=, <, and <= on numeric and date types using the format:

field operator value [and field operator value]

For example:
error >= 400 and error <=500
o_totalprice > 130000

In a Lucene search, the range format is different. For example:

error:[400 TO 500]
o_totalprice:[130000 to 140000]

Regular expressions

In a Lucene search, the forward slash / / characters can be used to start and end a regular expression search string as a free search or for a field search. For example, searching for /[aA]venue/ will return results that contain either avenue or Avenue. Regular expressions are based on the Lucene library for expressions, and might be different from other general regex libraries and format.

Histogram Display and Time Settings

At the top of the search results page is a histogram of the results over the query time range. The histogram is displayed when the Refinery view has a defined timestamp field (selected when the view was created). Some files might not have timestamps—those files are typically dimension tables with fields that contain supporting information. Queries against views that do not have a timestamp will not display the histogram, nor the time range selector (since it is meaningless for those tables). As an example, region-view contains information with no timestamp details – a dimension table. If you query that view, the histogram and time ranges do not appear.

13751375

Setting Time Ranges

The default time range for a search is the last 15 minutes. It is very easy to change and adjust the time ranges using the Kibana controls to select other ranges that better suit the index data or the type of investigation that you are performing.

For example, you can click the calendar icon next to the time range value to see a pop-up window with some easy-to-select options including an adjustment for the last number of minutes, or for selecting from commonly used values, or by selecting from the range values used in recent searches:. There is also a Refresh option to update the data on a scheduled time frame, which can be helpful for live object groups and the continuously new data that they receive. (Refresh is not very useful for searches on static indexes, because as the name implies, that data typically is not changing over time.)

14301430

You can click the starting or ending time range to display another set of time options to set the start or end time value using a calendar widget or a relative interval widget, or to set a time to "now" using a quick click.

14481448

Alternatively, the histogram shows a window of data over the current time period. Click and drag inside of the histogram to zoom to a more specific duration of time. The time picker updates for the selected range, and the histogram updates to show the new data points.

13441344

Query Your Data

The power of ChaosSearch is the ability to effectively transform S3 objects into searchable entities. You can type Kibana Query Language (KQL) into the search bar above the histogram to refine the results.

Enter your desired search criteria into the search bar and press Enter.

28382838

👍

Lucene Query Syntax

Elasticsearch uses the Lucene query syntax for searches. Refer to query string syntax for more information about constructing your queries.

Explore File Details

Click the expand button (carat) on the left of any row in the search results to view all file data as a table.

16701670

By default, the data is displayed in a tabular format. To view the data as a JSON object, click the JSON tab.

26262626

Save a Query

Saving queries allows you to name and quickly reload the query and any filters in the Discover page. You can use the saved query to quickly load a favorite visualization.

To save a query:

  1. Click the disk icon. The Saved Queries window opens.
896896
  1. Click Save current query. The Save query window opens.
914914
  1. Type a name for the query. This example uses AppCost as the name. You can also choose to add a description string, and whether to save the filters and the time range used for the query that you are saving.
  2. Click Save.

The query is added to the list of queries that appears when you click the carat next to the disk icon.

886886

You can delete a saved query by clicking the disk icon to open the list, and then clicking the trashcan icon to delete the saved query. You will be prompted to confirm the deletion.

11761176

Filtering the Record Display

You can use the Available fields selections in the left list to reduce and tune the data that appears for the records returned below the histogram. For example, you could select o_clerk and o_orderpriorty to show only those two columns in addition to the timestamp (when time is part of the view):

17121712

The filter fields that you select can be cleared by hovering over each one and clicking the x to remove that option from the Selected fields area on the left. If you remove all of the selected options, the record display returns to its complete form.

👍

More information

See the Kibana documentation for more information about exploring your data using Discover.


What’s Next

Navigate to the Visualization screen

Did this page help you?