ChaosSearch has the unique ability to define easy, virtual data transformations. Data analysts can use field type overrides to create a virtual mapping of a source data type to a better virtual type for indexing and analysis.
In the source data files, it is not unusual for data to have a format where the actual value (or content) is not clearly aligned to its purpose or meaning. For example, IDs are typically string values, but sometimes they are numeral values like 12345. If a numeric ID field defaults to a number type, it is possible that ID values could be used incorrectly as a number, when they should be used as strings for indexing and analysis.
Because ChaosSearch supports virtual typing, type overrides are defined in the ChaosSearch indexing and analytics configuration. Data analysts can use object groups and overrides to improve how to classify the raw data without the time and expense to re-write or re-assign types in the logging pipeline.
After you have selected a storage bucket, started the process to create an object group, and created the filters for the objects to include, you can then review and override any field within the object.
To create an object type override:
- In the object group content preview window, click Schema Filter in the top right corner.
- In the Column Overrides dialog, click Add Override to add an override row with a Column and Type field.
- Type the value for the Column that you want to override and select the new Type from the drop-down list.
- Click Add Override to add another editable row if needed, and repeat the process to select a column and type.
- Click Submit to save the overrides.
Once the object group is created, the new Type that you assigned appears in the group content section. In the following example, an override was used to change a port value from a number to a string for indexing.
Updated 5 months ago