Run distributed training with the SageMaker AI distributed data parallelism library (original) (raw)

The SageMaker AI distributed data parallelism (SMDDP) library extends SageMaker training capabilities on deep learning models with near-linear scaling efficiency by providing implementations of collective communication operations optimized for AWS infrastructure.

When training large machine learning (ML) models, such as large language models (LLM) and diffusion models, on a huge training dataset, ML practitioners use clusters of accelerators and distributed training techniques to reduce the time to train or resolve memory constraints for models that cannot fit in each GPU memory. ML practitioners often start with multiple accelerators on a single instance and then scale to clusters of instances as their workload requirements increase. As the cluster size increases, so does the communication overhead between multiple nodes, which leads to drop in overall computational performance.

To address such overhead and memory problems, the SMDDP library offers the following.

• The SMDDP library optimizes training jobs for AWS network infrastructure and Amazon SageMaker AI ML instance topology.
• The SMDDP library improves communication between nodes with implementations ofAllReduce and AllGather collective communication operations that are optimized for AWS infrastructure.

To learn more about the details of the SMDDP library offerings, proceed to Introduction to the SageMaker AI distributed data parallelism library.

For more information about training with the model-parallel strategy offered by SageMaker AI, see also (Archived) SageMaker model parallelism library v1.x.

Topics

• Introduction to the SageMaker AI distributed data parallelism library
• Supported frameworks, AWS Regions, and instances types
• Distributed training with the SageMaker AI distributed data parallelism library
• Amazon SageMaker AI data parallelism library examples
• Configuration tips for the SageMaker AI distributed data parallelism library
• Amazon SageMaker AI distributed data parallelism library FAQ
• Troubleshooting for distributed training in Amazon SageMaker AI
• SageMaker AI data parallelism library release notes

Scaling training

Introduction to the SMDDP library

Did this page help you? - Yes

Thanks for letting us know we're doing a good job!

If you've got a moment, please tell us what we did right so we can do more of it.

Did this page help you? - No

Thanks for letting us know this page needs work. We're sorry we let you down.

If you've got a moment, please tell us how we can make the documentation better.