Cloud Storage File Best Practices

Some recommendations for planning and organizing your cloud storage buckets and files

Aggregation Bucket Organization

Log files can have a variety of formats and come from different sources. In ChaosSearch, the most efficient indexes are those with logs that have a common structure that can be grouped into virtual buckets called object groups. To ease the object group creation, it is a good practice to organize log files with common filename prefixes in cloud storage. Object group creators can use filename prefix rules to collect the same log files into object groups and index them together. An intuitively organized cloud storage bucket also helps bucket administrators to quickly recognize the data that resides in it.

While not all of these categories are relevant to all customers or business units, some useful factors to consider when organizing logs in cloud storage are the following:

  • Log structure (CSV, JSON, custom formats)
  • Log type (application, metrics)
  • Application Types (Apache, NGINX, loadbalancer)
  • Microservice Type
  • Region
  • AccountID
  • Log date

For example, you might find log files with the following names in a well organized bucket. In this case, it would be very easy to create indexes that cover only application1, application1 in a specific region, application1 for a specific month, and so on, for each application (application2, ...):

/applogs/application1/us-east-1/2021/03/25/logname1.json
/applogs/application1/us-east-1/2021/03/25/logname2.json
/applogs/application1/us-west-1/2021/03/25/logname1.json
/applogs/application2/us-east-1/2021/03/25/logname.json

Data Retention

The retention of ChaosSearch indexed data is configured through the ChaosSearch user interface. There are no requirements for additional object storage lifecycle policies to expire old data. Once indexed, business units can delete data or use intelligent tiering policies to move objects to less active (and often less expensive) tiers of storage.


Did this page help you?