This guide lists the steps required to send Cloudwatch logs to S3. It involves:

1. Creating a shipper lambda which uploads Cloudwatch logs to S3 and its corresponding IAM roles and policies.
   • Sample outputs of IAM roles and policies
2. Cloudwatch event rules to trigger the shipper lambda against user specified Cloudwatch logs.
   • Sample Cloudwatch event rule

Pre-requisites

1. Credentials which have access to do the following operations in the target AWS account.

   1. Create/Update AWS IAM roles, policies
   2. Create/Update AWS S3 buckets
   3. Create/Update AWS Lambda
   4. Create/Update AWS Cloudwatch event rules

2. Following commands

   1. aws - installation instructions
   2. wget
   3. jq - installation instructions

3. Last9 monitoring user ARN - created during Last9 onboarding

   export last9_monitoring_user_arn=xyz
   

4. Create an S3 bucket to which Cloudwatch will send logs.

5. Enable read-only access to the target S3 bucket by following these instructions.

Deploy Cloudwatch to S3 Lambda

1. Download the infra creation script

   cd /tmp/
   wget <https://last9-custom-metrics.s3.ap-south-1.amazonaws.com/stable/create-cloudwatch-to-s3-lambda-infra.sh>
   chmod +x create-cloudwatch-to-s3-lambda-infra.sh
   

2. Run the script in dry run mode to verify what infra will be created

   # replace $target_s3_bucket by bucket to which you want to send logs
   DRYRUN=1 ./create-cloudwatch-to-s3-lambda-infra.sh $target_s3_bucket
   
   

   • Sample output

3. Run the script without dry run mode to create the above infra

   DRYRUN=0 ./create-cloudwatch-to-s3-lambda-infra.sh  $target_s3_bucket | tee /var/tmp/cloudwatch-to-s3-lambda-infra.log 2>&1
   
   

   • Sample output

4. Last line of output from the above creation on success will be

   STATUS: lambda: Created use arn:aws:lambda:ap-south-1:xxxx:function:cloudwatch_to_s3
   
   

5. Save the arn in an enviornment variable

   export lambda_arn=arn_output_from_above_step
   

6. Re-run the script to grant permissions to last9 user to get, deploy and invoke the lambda

   
   DRYRUN=0 ./create-cloudwatch-to-s3-lambda-infra.sh $target_s3_bucket $last9_monitoring_user_arn
   

Cloudwatch event rule to ship logs to S3

Once the above Lambda is deployed, you can use this section to schedule it to ship logs of multiple log groups by adding a Cloudwatch event rule per log group.

1. Use the Lambda ARN in the previous section to add Cloudwatch event rule to trigger itself.

   cd /tmp/cloudwatch_to_s3/files
   ./create-lambda-event-rule.sh LambdaLogsToCloudwatch $lambda_arn
   

   • Sample output