Intervals

When you define an object group to specify the cloud-storage files that you want to index and the rules for indexing them, and then start indexing, the Chaos Index service creates one or more daily intervals. The intervals are listed in the Intervals pane for object groups and are also used in views.

Intervals are very important components of the ChaosSearch environment. This topic provides a closer look at intervals, what they do, and how they are managed within the ChaosSearch ecosystem.

How are Intervals Created and Named

Intervals are created by the Chaos Index service, and are based on two key inputs:

• The object group that defines your raw cloud storage files to index, and the rules to index them (such as isolation keys)
• The creation date (as recorded in cloud storage) of the raw storage files indexed by the group

When ChaosSearch starts to index an object group, it searches the defined cloud storage locations (in the object group) to find the matching files to index. For each matching file, it captures the creation date for the file and indexes its contents to create the patented, lossless index data used by the ChaosSearch Refinery for queries and analytics.

Using the object group name, and the creation date of the cloud storage file, ChaosSearch creates daily intervals to organize the indexed data. Each daily interval contains the indexed data and related information for all of the matching files with the same creation day. A sample Intervals page for an object group follows:

As shown in the example, interval names combine the object group name and the creation day/date of the files indexed by the object group, for example:

_cloudtrail_data_2022-10-04_

This is the daily interval for the cloudtrail_data object group with the indexed log and event files that have a creation day of October 4, 2022.

Keep in mind that the indexed data for your cloud storage object files is stored in the read-write cloud storage bucket that you own. You can use the interval files to manage the analysis of and lifespan of the indexed data and its related metadata.

📆

Cloud storage file creation dates and timestamps inside the log and event files

The creation date for a log or event file written to cloud storage is usually close to or the same as the timestamps captured for the events and log entries inside the file. However, sometimes the creation date could be later than the internal timestamps.

The daily interval file names use the creation date, not the date of timestamps inside the files.

As an example, a log file for events that occurred on January 1, 2024 could have a later cloud storage creation date such as March 15, 2024. When ChaosSearch indexes the file, the date used for the daily interval file name will be 2024-03-15 (the file creation date). Queries against the view show the January 1, 2024 timestamps for the events and records.

Interval Lifecycle and Retention

ChaosSearch's compact indexing design enables users to keep their indexed data for a very long time, even indefinitely if needed, with lower storage costs compared to data retention for other applications. The timeline of the analysis that you want to keep is up to you.

In many cases, customers will eventually age out the bulky, original, cloud source data indexed by ChaosSearch; they could possibly delete it, or move older content to less-expensive long term archival storage. They could then keep the more compact and lossless Chaos Index data in accessible storage for longer periods to enable historical analysis and querying.

The daily intervals are the key to that historical data retention. When you create an object group, the Retention Policy setting specifies how long to keep the indexed data. The data clean-up process uses the date in the daily interval name to identify the indexed data to delete. As an example, if an object group uses the default retention policy of 14 Days, ChaosSearch automatically cleans up and removes any daily intervals for that object group with an interval file name date component that is prior to the two-week period. That is, on October 14, 2022, the object group daily intervals _<object-group-name>_2022-09-30_ and earlier would be deleted in the clean-up process.

❗️

Be careful when changing (especially reducing) retention periods for daily intervals.

Always use caution when decreasing the retention period for an object group. If a group changes from Unlimited retention to 3 Days, for example, the processing work to delete a potentially large number of daily intervals could impact system performance. Also, you cannot restore the deleted daily intervals except by re-indexing the original cloud storage files.

If the object group has an Unlimited retention period, the object group's daily intervals are never automatically deleted.

Administrators can delete daily intervals manually from the Storage > Intervals page. Typically, daily intervals are manually deleted when the administrator wants to clean up some old intervals, or when the object group is going to be deleted. (You must delete all daily intervals for an object group before you can delete the object group.) Sometimes, stale or unused intervals might be deleted as part of a trial and setup process with ChaosSearch Customer Success while changing and re-indexing an object group for testing.

Daily Intervals and Views

When you create a Chaos Refinery view to query and analyze the indexed data for one or more object groups, part of the view definition is the list of daily intervals to include in the view. ChaosSearch offers options to select all intervals, or a subset, to refine the query analysis scope. For example, one could select:

• One, more, or all of the daily intervals
• Daily intervals that match an interval name pattern specified by a regular expression
• A rolling time window of daily intervals, such as those for the last 7 days, last 45 days, or last 12 months

The interval pattern and window both match on the daily interval name to determine which intervals to include in the view (and thus in the analysis).