Field Type Overrides

Object groups support field data type overrides that can refine how source content is indexed and made available for analytics.

ChaosSearch uses an auto-detect feature to identify the data types for the fields in the files that you have selected for indexing. Auto-detection scans each file and selects a data type—such as string, number, timeval, or period—that best classifies the format for the field in the indexed data.

Data typing for a field could change if new files for that object group contain data that is a different type than in previous files. For example, a field that previously held number data could change to a string in a later index segment if the auto-detection finds alphanumeric data in that field. The alphanumeric data could represent an expected change in content for the indexed files of an object group, or it could be an unexpected change/problem for the files.

When you create a Refinery view for an object group, you select a timeval field (if one is available in the data) to be the source for time-based displays such as Discover histograms. If that timeval field changes to a different data type such as string, any Refinery view that uses that field for its timestamp will return errors in Discover histograms for time periods when the field is classified as a string.

When a field changes type or has special type circumstances, analytics that reference the field are affected. For example, a number field that changes to a string will have a different display format, and the new data type for that field could impact visualizations or aggregations that relied on that field with its prior type. If your data includes a value that is a numeric identifier, like a customer ID or numeric serial number, you might want to treat those number values as strings for analysis. You can override a numeric identifier field to be indexed as a string with the column overrides feature.

ChaosSearch supports the ability to override the auto-detection data type for a field. As ChaosSearch indexes new files for an object group, the overrides defined for fields in that object group will be used even when the content of that field might be auto-detected as a different data type. In the timestamp field case, a field that has a configured override to timeval would continue to have a timeval data type, even when the field content is not detectable as a a valid timestamp value.


Use Caution with Data Type Overrides

Avoid coercing the data type of object group fields unless necessary or as directed by Customer Success or Engineering. Data type overrides affect the storage and nature of the indexed data. You cannot change an object group to alter or remove an override. It is highly recommended to carefully plan overrides after a careful review of the raw source files and the fields that are used for analysis.

Field Override

During the object group creation process, after you have selected a storage bucket and defined the expressions for the cloud storage objects to include, you can use the Schema Filter button to define a data type override if needed. Make sure that you know the field name and the desired data type before you proceed.

To create a field type override:

  1. In the object group content preview window, click Schema Filter in the top right corner.
  1. In the Column Overrides dialog, click Add Override to add an override row with a Column and Type field.
  1. In the Column field, type the source file field name with the data type that you want to override.
  2. In the Type field, select the data type override from the drop-down list. The default is String.
  3. Optionally, if you have another override to define, click Add Override to add another editable row. Specify the field name and the override data type that you want to assign.
  4. Click Submit to save the overrides.

When the object group is created, the new Type that you assigned appears in the group content section. In the following example, an override was used to change a port value from a number data type to a string for indexing and analysis.


Did this page help you?