Schema Transformations

You can use the view Schema Transformations to virtually transform and materialize columns for analytics and querying.

When creating a view, you can virtually transform the schema of any of the fields within the indexed data structure. The ChaosSearch virtual transformations provide powerful features to tailor the columns in the resulting schema while leaving the source indexed data intact and available for different transformations with other views for other users or purposes.

To create a transformation:

  1. While in the create view process, and in the Schema Transformation window, select or search for the field that you want to transform.
  2. Click the gear icon at the right end of the field row. The Schema Transformation window opens.
  1. In the Schema Transformation window, you can select from several predefined transformations.
  2. Select the desired transformation type, and supply any additional field content as applicable, then click Save Transform.

The transformation options include the following:



Materialize with Regex

Use a custom regular expression to transform a field, perhaps into several searchable columns with distinct content.

Materialize with JSON

Use a JSONPath expression to select and transform a field inside a JSON string blob as a materialized column that can be used for filtering and analytics.

Treat as IP

For an IP address that is usually stored as a string value, select Treated as IP to virtually transform that field to an IP data column.

Treat as GeoPoint

For geo-location data, treat the content of the indexed data field as longitude or latitude geolocation data.

Treat as Partition Key

For ChaosSearch object groups that use isolation keys to split the indexed data into tenant-specific or other specific chunks, treat this column as the filtering key that controls which chunk(s) of data to include as part of the search results or analytics of this view.

The Treat as IP, Treat as GeoPoint, and Treat as Partition Key transforms essentially set the data type for the column to the specified data type for the purposes of visualization and analysis. The Materialize transforms are more powerful and customizable features, described in the following sections.

Regular Expression Transformations

As an example of a Materialize with Regex transformation, if you have a log file with a field that contains a string of web URL data, you might want to transform the field into three virtual/materialized columns that each contain a specific part such as the domain, port, and path components for use with analysis or filtering purposes.

After selecting the URL field and clicking the gear icon to transform it:

  1. Make sure that Materialize with Regex is selected as the transformation.
  2. Type the regular expression pattern to use for each field, such as (\S+[ :])(\d+)(\S+) as an example. See Refinery Transformation Regex for other sample patterns.
  3. Click Add Field to add three fields, and name them domain, port, and path. Make sure that domain and path are STRING types, while port is a NUMBER type.
  4. Click Refresh to update the Preview pane and review the transformation. If there are any errors or changes, you can update the regular expression and/or fields and refresh again.
  5. Click Save Transform when finished.

After saving your transformation changes, the Schema Transformation window shows the transformed fields as in the following example:


JSON Transformations

If an indexed data field contains JSON string content, you can use this transformation to specify one or more specific JSON properties as a materialized column for analytics. In many implementations, JSON string content supports only text searching within their content, but the ChaosSearch Materialize with JSON transformation allows you to specify a JSON path value to select a property within the JSON string, and create a materialized column for the view. The JSON string blob remains intact for text searches and other analysis.

After selecting the JSON string field and clicking the gear icon to transform it:

  1. Make sure that Materialize with JSON is selected as the transformation.
  2. Click Add Field to create a placeholder for each field that you want to transform.
  1. For each field, type the JSONPath expression pattern to identify each property that you want to materialize as a column. You can use third-party tools such as the online JSON path formatter to work out the required JSON path value. In this example, a simple JSNPath is used to isolate the connect:version property as a new column named version.
  2. Click Refresh to update the Preview pane and review the transformation. If there are any errors or changes, you can update the path expression and/or fields and refresh again.
  3. Click Save Transform when finished. The new virtual column shown on the Schema Transformation window.

Did this page help you?