Index Lifecycle Management

ChaosSearch Index Lifecycle

The Chaos Index process creates daily intervals when it indexes the cloud storage objects for an object group, and it saves the daily intervals in your designated storage bucket. These daily intervals contain the indexed data that is used by the ChaosSearch features for queries, visualizations, and analytics.

Lifecycle management allows administrators to define how long to keep daily intervals (and related indexed data) for each object group to automate the retention and cleanup. You can define the retention period to set an age-out time to control when daily intervals are removed.

🚧

NOTE:

The aging process uses the date in the filename of the daily interval, which is taken from the modification date for the cloud storage files indexed in that interval for an object group.

Setting a Retention Policy

Lifecycle management is defined at the end of object group creation, as described in Creating Object Groups. Administrators can select the Retention Policy and define when daily intervals should be deleted. Time frames can be specified as a number of days or months.

Administrators have the option to retain an object group's daily intervals "forever" without an automated cleanup. Deselect the Retention Policy field to specify an Unlimited retention of daily intervals.

Modifying the Retention Policy

After an object group is created, administrators can modify the retention policy settings to adjust the lifecycle for the daily intervals. Changes to the policy can either increase or decrease the aging out of the indexes associated with the object group.

❗️

Use caution when decreasing the retention policy value.

If you decrease the Retention Policy below its current value, the system immediately deletes the daily interval files that have a filename date before the retention time period. Deleted intervals cannot be recovered; they must be recreated by re-indexing the original cloud storage files.

To modify the interval lifecycle setting:

1. In the Storage area, select the object group for which you want to change the interval lifecycle.
2. Click the Properties tab.
3. Click the current value for the Retention Policy. The Retention Policy window opens.

4. Specify the new desired time period for the policy. You can select a new number of days/months as applicable. You can change to/from Unlimited retention as needed. When Unlimited is selected, you cannot edit the Overall number and Time Unit fields.
5. Click View Full Retention Policy to display the retention settings as a JSON file. For example:

6. Click Save to place the new retention period into effect.

When the Retention Policy range is updated, the system displays a confirmation window when the changes could result in the deletion of daily intervals that have dates before the new retention period value. Click OK to confirm that the new retention period is correct and that any daily intervals for prior days should be removed.

The interval cleanup is a process that runs on a periodic basis to remove intervals that have aged beyond the group's retention policy time. It can take several minutes before old intervals are removed.

Checking Daily Interval Cleanup Events

When ChaosSearch runs a cleanup to remove old daily intervals, a message is added to the Events tab of the object group stating that the _interval_file_ exceeded the interval retention.

If you have a time-limited retention policy, you can check the Events tab to confirm that the system is removing old daily intervals that are earlier than the retention period.