Use CloudWatch Logs to monitor AWS Batch on Amazon EKS jobs (original) (raw)

PrerequisitesInstall the add-on

You can use Amazon CloudWatch Logs to monitor, store, and view all your log files in one location. Using CloudWatch Logs, you can search, filter, and analyze log data from multiple sources.

You can download an AWS for Fluent Bit image that includes a plugin to monitor AWS Batch on Amazon EKS jobs in CloudWatch Logs. Fluent Bit is an open-source log processor and forwarder that's both Docker and Kubernetes compatible. We recommend that you use Fluent Bit as your log router because it's less resource intensive than Fluentd. For more information, see Install the CloudWatch agent with the Amazon CloudWatch Observability EKS add-on or the Helm chart.

Prerequisites

• Attach the CloudWatchAgentServerPolicy policy to the AWS Identity and Access Management policy of your worker nodes. For more information, see Verify prerequisites.

Install the add-on

For instructions on how to install AWS for Fluent Bit and create the CloudWatch groups, see Install the CloudWatch agent with the Amazon CloudWatch Observability EKS add-on or the Helm chart.

You must provide the following additional configuration data when installing the add-on:

• If you install the add-on with the AWS Management Console you need to provide the following tolerations inConfiguration values:

{  
  "tolerations": [  
    {  
      "key": "batch.amazonaws.com/batch-node",  
      "operator": "Exists"  
    }  
  ]  
}  

• If you install the add-on with the AWS CLI then add the following arguments:

--configuration-values '{"tolerations":[{"key":"batch.amazonaws.com/batch-node","operator":"Exists"}]}'  

Tip

Remember that Fluent Bit uses .5 CPU and 100 MB of memory on AWS Batch nodes. This reduces the total available capacity for AWS Batch jobs. Consider this when you size your jobs.

Tutorial: Turn on Container Insights

Tag your resources

Did this page help you? - Yes

Thanks for letting us know we're doing a good job!

If you've got a moment, please tell us what we did right so we can do more of it.

Did this page help you? - No

Thanks for letting us know this page needs work. We're sorry we let you down.

If you've got a moment, please tell us how we can make the documentation better.