Field Overrides

ChaosSearch uses an auto-detect feature to identify the fields and their data types within the files that you select for indexing. Auto-detection scans each file as it is indexed, and selects a data type—such as string, number, timeval, or period—to classify the format for the field in the indexed data.

📘

Timeval and Period Data Types

A period data type is a string representation of date/time (e.g., 2018-02-13 20:03:53.589918) and is not converted into a number representation. Timeval is the number version of date/time based on milliseconds from Epoch (that is, January 1st, 1970 at 00:00:00 UTC). This is also referred to as UNIX system time.

In most cases, it is best to use a Timeval data type because Timeval is more performant for most operations in both time and space. The Period data type is helpful for the less common case where there is an advantage to "keep" a column as a string data type to know and preserve the format of the raw source.

You can override the auto-detected data type for a field. As ChaosSearch indexes new files for an object group, the overrides will be used even if the content of that field is auto-detected as a different type. In the timestamp field case, a field that has a configured override to timeval would keep that timeval data type, even if the field content is not detectable as a a valid timestamp value.

❗️

Use Caution with Data Type Overrides

Avoid coercing the data type of object group fields unless necessary or as directed by Customer Success or Engineering. Data type overrides affect the storage and nature of the indexed data. You cannot change an object group to alter or remove an override. It is highly recommended to carefully plan overrides after a careful review of the source files and the fields that are used for analysis.

Setting a Data Type Override

During the object group creation process, after you have selected a storage bucket and defined the expressions for the cloud storage objects to include, you can use the Field Override button to define a data type override if needed. Make sure that you know the field name and the desired data type before you proceed.

To create a field override:

1. In the object group content preview window, click Field Override in the top right corner.

2. In the Field Overrides dialog, click Add Override to add an editable row with a Field and Data Type value.

3. For Field, type the field name that you want to override.
4. In the Data Type field, select the data type override from the drop-down list. The default is String.
5. Optionally, if you want to define another override, click Add Override to add another row and specify the field and data types.
6. Click Save to save the overrides.

When the object group is created, the data types that you assigned appear in the Properties section.

📘

Data Types Could Change in New Daily Intervals

If you do not specify a field override, the data type for a field could change if new files for that object group contain data that is a different type versus previous files. Sometimes the source application that creates log and event data has a change in schema. For example, a field that previously held a string could change to a number/enumerated value. ChaosSearch will update the daily intervals for the change in field data types, but this could have an impact to your views and any aggregations or visualizations that use those fields.

When you create a Refinery view for an object group, you select a timeval field (if one is available in the data) to be the source for time-based displays such as Discover histograms. If that timeval field changes to a different data type such as a string, any Refinery view that uses that field for its timestamp will return errors in Discover histograms for time periods when the field is classified as a string.