Archiving Datadog Log Files

As you plan the Datadog-ChaosSearch integration, make sure that any log files that you monitor with Datadog are archived to cloud-storage.

• If you are already archiving the log files, the integration begins by granting ChaosSearch to index those archive buckets. For related information, see Pushing Logs to Cloud Storage.
• If you are not currently archiving the log files, start archiving them to cloud storage by using the Datadog archive feature as described in the next section, or via other log shippers that you might already have in place.

ChaosSearch works with Amazon Web Services cloud resources, as well as Google Cloud Platform. For brevity, the following sections show specific steps for AWS, with links to details for GCP when needed.

Configure Datadog to Archive Log Files

This section describes how to set up a log archive in Datadog if log files are not already backed up in cloud storage. More logging information is available in the Datadog Log Management documentation, and the detailed steps for the process to create an archive to cloud storage is in the Datadog Log Archives page. You can also contact ChaosSearch Customer Success for assistance with this procedure.

To complete this step, you must have access to a Datadog account with the logs_write_archive permission to manage log archive configurations.

1. Follow the steps in the Datadog Log Archives page to set up an archive for forwarding ingested logs to your own cloud-hosted storage bucket, such as an S3 bucket.
2. In the "Route your logs to a bucket" section of that process, you log in to the Datadog dashboard UI.
   1. In the left Datadog menu, select Logs > Configuration to open the Pipelines page, then click the Archives tab (https://app.datadoghq.com/logs/pipelines/archives).
      
   2. In the Archives page, click New Archive to create a new archive:
      
   3. In the Add an Archive window, complete the information to associate the archive with the cloud-store bucket and policy/permissions that you created in Step 1. An example screen follows with sample text:
      

After the cloud-store bucket and Datadog archive setup is completed, log files will start landing in the designated bucket.

Make sure that you grant the ChaosSearch role for your site access to the archive bucket, as described in the ChaosSearch topic Configure Cloud Storage Access (and for S3, the AIM Role Policy Permissions section).