Previous Releases Notes (Neuron 2.x) — AWS Neuron Documentation (original) (raw)

Previous Releases Notes (Neuron 2.x)#

Table of contents

• Neuron 2.22.1 (05/12/2025)
• Neuron 2.22.0 (04/03/2025)
• Neuron 2.21.1 (01/14/2025)
• Neuron 2.21.0 (12/20/2024)
• Neuron 2.21.0 Beta (12/03/2024)
• Neuron 2.20.2 (11/20/2024)
• Neuron 2.20.1 (10/25/2024)
• Neuron 2.20.0 (09/16/2024)
• Neuron 2.19.1 (07/19/2024)
• Neuron 2.19.0 (07/03/2024)
• Neuron 2.18.2 (04/25/2024)
• Neuron 2.18.1 (04/10/2024)
• Neuron 2.18.0 (04/01/2024)
• Neuron 2.17.0 (02/13/2024)
• Neuron 2.16.1 (01/18/2024)
• Neuron 2.16.0 (12/21/2023)
• Neuron 2.15.2 (11/17/2023)
• Neuron 2.15.1 (11/09/2023)
• Neuron 2.15.0 (10/26/2023)
• Neuron 2.14.1 (09/26/2023)
• Neuron 2.14.0 (09/15/2023)
• Neuron 2.13.2 (09/01/2023)
• Neuron 2.13.1 (08/29/2023)
• Neuron 2.13.0 (08/28/2023)
• Neuron 2.12.2 (08/19/2023)
• Neuron 2.12.1 (08/09/2023)
• Neuron 2.12.0 (07/19/2023)
• Neuron 2.11.0 (06/14/2023)
• Neuron 2.10.0 (05/01/2023)
• Neuron 2.9.1 (04/19/2023)
• Neuron 2.9.0 (03/28/2023)
• Neuron 2.8.0 (02/24/2023)
• Neuron 2.7.0 (02/08/2023)
• Neuron 2.6.0 (12/12/2022)
• Neuron 2.5.0 (11/23/2022)
• Neuron 2.4.0 (10/27/2022)
• Neuron 2.3.0 (10/10/2022)

Neuron 2.22.1 (05/12/2025)#

Neuron 2.22.1 release includes a Neuron Driver update that resolves DMA abort errors on Trainium2 devices. These errors were previously occurring in the Neuron Runtime during specific workload executions.

Neuron 2.22.0 (04/03/2025)#

Table of contents

• What’s New

What’s New#

The Neuron 2.22 release includes performance optimizations, enhancements and new capabilities across the Neuron software stack.

For inference workloads, the NxD Inference library now supports Llama-3.2-11B model and supports multi-LoRA serving, allowing customers to load and serve multiple LoRA adapters. Flexible quantization features have been added, enabling users to specify which model layers or NxDI modules to quantize. Asynchronous inference mode has also been introduced, improving performance by overlapping Input preparation with model execution.

For training, we added LoRA supervised fine-tuning to NxD Training to enable additional model customization and adaptation.

Neuron Kernel Interface (NKI): This release adds new APIs in nki.isa, nki.language, and nki.profile. These enhancements provide customers with greater flexibility and control.

The updated Neuron Runtime includes optimizations for reduced latency and improved device memory footprint. On the tooling side, the Neuron Profiler 2.0 (beta) has added UI enhancements and new event type support.

Neuron DLCs: this release reduces DLC image size by up to 50% and enables faster build times with updated Dockerfiles structure. On the Neuron DLAMI side, new PyTorch 2.5 single framework DLAMIs have been added for Ubuntu 22.04 and Amazon Linux 2023, along with several new virtual environments within the Neuron Multi Framework DLAMIs.

More release content can be found in the table below and each component release notes.

For detailed release artificats, see Release Artifacts.

Neuron 2.21.1 (01/14/2025)#

Neuron 2.21.1 release pins Transformers NeuronX dependency to transformers<4.48 and fixes DMA abort errors on Trn2.

Additionally, this release addresses NxD Core and Training improvements, including fixes for sequence parallel support in quantized models and a new flag for dtype control in Llama3/3.1 70B configurations. See NxD Training Release Notes (neuronx-distributed-training) for details.

NxD Inference update includes minor bug fixes for sampling parameters. See NxD Inference Release Notes.

Neuron supported DLAMIs and DLCs have been updated to Neuron 2.21.1 SDK. Users should be aware of an incompatibility between Tensorflow-Neuron 2.10 (Inf1) and Neuron Runtime 2.21 in DLAMIs, which will be addressed in the next minor release. See Neuron DLAMI Release Notes.

The Neuron Compiler includes bug fixes and performance enhancements specifically targeting the Trn2 platform.

Neuron 2.21.0 (12/20/2024)#

Table of contents

• What’s New
• 2.21.0 Known Issues and Limitations

What’s New#

Overview: Neuron 2.21.0 introduces support for AWS Trainium2 andTrn2 instances, including the trn2.48xlarge instance type and Trn2 UltraServer (Preview). The release adds new capabilities in both training and inference of large-scale models. It introduces NxD Inference (beta), a PyTorch-based library for deployment, Neuron Profiler 2.0 (beta), andPyTorch 2.5 support across the Neuron SDK, and Logical NeuronCore Configuration (LNC) for optimizing NeuronCore allocation. The release enables Llama 3.1 405B model inference on a single trn2.48xlarge instance.

NxD Inference: NxD Inference (beta) is a new PyTorch-based inference library for deploying large-scale models on AWS Inferentia and Trainium instances. It enables PyTorch model onboarding with minimal code changes and integrates with vLLM. NxDI supports various model architectures, including Llama versions for text processing (Llama 2, Llama 3, Llama 3.1, Llama 3.2, and Llama 3.3), Llama 3.2 multimodal for multimodal tasks, and Mixture-of-Experts (MoE) model architectures including Mixtral and DBRX. The library supports quantization methods, includes dynamic sampling, and is compatible with HuggingFace checkpoints and generate() API. NxDI also supports distributed strategies including tensor parallelism and incorporates speculative decoding techniques (Draft model and EAGLE). The release includes Llama 3.1 405B model sample, Llama 3.3 70B model sampleand Llama 3.1 405B model with speculative decoding for inference on a single trn2.48xlarge instance.

For more information, see NxD Inference documentation and check the NxD Inference Github repository: aws-neuron/neuronx-distributed-inference

Transformers NeuronX (TNx): This release introduces several new features, including flash decoding support for speculative decoding, and on-device generation in speculative decoding flows. It adds Eagle speculative decoding with greedy and lossless sampling, as well as support for CPU compilation and sharded model saving. Performance improvements include optimized MLP and QKV for Llama models with sequence parallel norm and control over concurrent compilation workers.

Training Highlights: NxD Training in this release adds support for HuggingFace Llama3/3.1 70B on trn2 instances, introduces DPO support for post-training model alignment, and adds support for Mixture-of-Experts (MoE) models including Mixtral 7B. The release includes improvedcheckpoint conversion capabilities and supports MoE with Tensor, Sequence, Pipeline, and Expert parallelism.

ML Frameworks: Neuron 2.21.0 adds support for PyTorch 2.5 and JAX 0.4.35.

Note

The CVEsCVE-2024-31583andCVE-2024-31580affect PyTorch versions 2.1 and earlier. Based on Amazon’s analysis, executing models on Trainium and Inferentia is not exposed to either of these vulnerabilities. We recommend upgrading to the new version of Torch-NeuronX by following the Neuron setup instructions.

Logical NeuronCore Configuration (LNC): This release introduces LNCfor Trainium2 instances, optimizing NeuronCore allocation for ML applications. LNC offers two configurations: default (LNC=2) combining two physical cores, and alternative (LNC=1) mapping each physical core individually. This feature allows users to efficiently manage resources for large-scale model training and deployment through runtime variables and compiler flags.

Neuron Profiler 2.0: The new profiler provides system and device-level profiling, timeline annotations, container integration, and support for distributed workloads. It includes trace export capabilities for Perfetto visualization and integration with JAX and PyTorch profilers, and support for Logical NeuronCore Configuration (LNC).

Neuron Kernel Interface (NKI): NKI now supports Trainium2 includingLogical NeuronCore Configuration (LNC), adds SPMD capabilities for multi-core operations, and includes new modules and APIs including support for float8_e5m2 datatype.

Deep Learning Containers (DLAMIs): This release expands support for JAX 0.4 within the Multi Framework DLAMI. It also introduces NxD Training, NxD Inference, and NxD Core withPyTorch 2.5 support. Additionally, a new Single Framework DLAMI for TensorFlow 2.10 on Ubuntu 22 is now available.

Deep Learning Containers (DLCs): This release introduces new DLCs for JAX 0.4 training and PyTorch 2.5.1 inference and training. All DLCs have been updated to Ubuntu 22, and the pytorch-inference-neuronx DLC now supports both NxD Inference and TNx libraries.

Documentation: Documentation updates include architectural details about Trainium2 and NeuronCore-v3, along with specifications and topology information for the trn2.48xlarge instance type and Trn2 UltraServer.

Software Maintenance: This release includes the following announcements:

• Announcing migration of NxD Core examples from NxD Core repository to NxD Inference repository in next release
• Announcing end of support for Neuron DET tool starting next release
• PyTorch Neuron versions 1.9 and 1.10 no longer supported
• Announcing end of support for PyTorch 2.1 for Trn1, Trn2 and Inf2 starting next release
• Announcing end of support for PyTorch 1.13 for Trn1 and Inf2 starting next release
• Announcing end of support for Python 3.8 in future releases
• Announcing end of support for Ubuntu20 DLCs and DLAMIs

Amazon Q: Use Q Developeras your Neuron Expert for general technical guidance and to jumpstart your NKI kernel development.

More release content can be found in the table below and each component release notes.

2.21.0 Known Issues and Limitations#

• See component release notes below for any additional known issues.

Neuron 2.21.0 Beta (12/03/2024)#

Note

This release (Neuron 2.21 Beta) was only tested with Trn2 instances. The next release (Neuron 2.21) will support all instances (Inf1, Inf2, Trn1, and Trn2).

For access to this release (Neuron 2.21 Beta), please contact your account manager.

This release (Neuron 2.21 beta) introduces support for AWS Trainium2 and Trn2 instances, including the trn2.48xlarge instance type and Trn2 UltraServer. The release showcases Llama 3.1 405B model inference using NxD Inference on a single trn2.48xlarge instance, and FUJI 70B model training using the AXLearn library across eight trn2.48xlarge instances.

NxD Inference, a new PyTorch-based library for deploying large language models and multi-modality models, is introduced in this release. It integrates with vLLM and enables PyTorch model onboarding with minimal code changes. The release also adds support for AXLearn training for JAX models.

The new Neuron Profiler 2.0 introduced in this release offers system and device-level profiling, timeline annotations, and container integration. The profiler supports distributed workloads and provides trace export capabilities for Perfetto visualization.

The documentation has been updated to include architectural details about Trainium2 and NeuronCore-v3, along with specifications and topology information for the trn2.48xlarge instance type and Trn2 UltraServer.

Use Q Developer as your Neuron Expert for general technical guidance and to jumpstart your NKI kernel development.

Note

For the latest release that supports Trn1, Inf2 and Inf1 instances, please see Neuron Release 2.20.2

Neuron 2.20.2 (11/20/2024)#

Neuron 2.20.2 release fixes a stability issue in Neuron Scheduler Extension that previously caused crashes in Kubernetes (K8) deployments. See Neuron K8 Release Notes.

This release also addresses a security patch update to Neuron Driver that fixes a kernel address leak issue. See more on Neuron Driver Release Notes and Neuron Runtime Release Notes.

Addtionally, Neuron 2.20.2 release updates torch-neuronx and libneuronxla packages to add support for torch-xla 2.1.5 package which fixes checkpoint loading issues with Zero Redundancy Optimizer (ZeRO-1). See PyTorch Neuron (torch-neuronx) release notes and Neuron XLA pluggable device (libneuronxla) release notes.

Neuron supported DLAMIs and DLCs are updated with this release (Neuron 2.20.2 SDK). The Training DLC is also updated to address the version dependency issues in NxD Training library. See Neuron DLC Release Notes.

NxD Training library in Neuron 2.20.2 release is updated to transformers 4.36.0 package. See NxD Training Release Notes (neuronx-distributed-training).

Neuron 2.20.1 (10/25/2024)#

Neuron 2.20.1 release addresses an issue with the Neuron Persistent Cache that was brought forth in 2.20 release. In the 2.20 release, the Neuron persistent cache issue resulted in a cache-miss scenario when attempting to load a previously compiled Neuron Executable File Format (NEFF) from a different path or Python environment than the one used for the initial Neuron SDK installation and NEFF compilation. This release resolves the cache-miss problem, ensuring that NEFFs can be loaded correctly regardless of the path or Python environment used to install the Neuron SDK, as long as they were compiled using the same Neuron SDK version.

This release also addresses the excessive lock wait time issue during neuron_parallel_compile graph extraction for large cluster training. See PyTorch Neuron (torch-neuronx) release notes and Neuron XLA pluggable device (libneuronxla) release notes.

Additionally, Neuron 2.20.1 introduces new Multi Framework DLAMI for Amazon Linux 2023 (AL2023) that customers can use to easily get started with latest Neuron SDK on multiple frameworks that Neuron supports. See Neuron DLAMI Release Notes.

Neuron 2.20.1 Training DLC is also updated to pre-install the necessary dependencies and support NxD Training library out of the box. See Neuron DLC Release Notes

Neuron 2.20.0 (09/16/2024)#

Table of contents

• What’s New
• 2.20.0 Known Issues and Limitations
• Neuron Components Release Notes
  • Inf1, Trn1/Trn1n and Inf2 common packages
  • Trn1/Trn1n and Inf2 only packages
  • Inf1 only packages

What’s New#

Overview: Neuron 2.20 release introduces usability improvements and new capabilities across training and inference workloads. A key highlight is the introduction of Neuron Kernel Interface (beta). NKI, pronounced ‘Nicky’, is enabling developers to build optimized custom compute kernels for Trainium and Inferentia. Additionally, this release introduces NxD Training (beta), a PyTorch-based library enabling efficient distributed training, with a user-friendly interface compatible with NeMo. This release also introduces the support for the JAX framework (beta).

Neuron 2.20 also adds inference support for Pixart-alpha and Pixart-sigma Diffusion-Transformers (DiT) models, and adds support for Llama 3.1 8B, 70B and 405B models inference supporting up to 128K context length.

Neuron Kernel Interface: NKI is a programming interface enabling developers to build optimized compute custom kernels on top of Trainium and Inferentia. NKI empowers developers to enhance deep learning models with new capabilities, performance optimizations, and scientific innovation. It natively integrates with PyTorch and JAX, providing a Python-based programming environment with Triton-like syntax and tile-level semantics, offering a familiar programming experience for developers. All of our NKI work is shared as open source, enabling the community developers to collaborate and use these kernels in their projects, improve existing kernels, and contribute new NKI kernels. The list of kernels we are introducing includes Optimized Flash Attention NKI kernel (flash_attention), a NKI kernel with an optimized implementation of Mamba model architecture (mamba_nki_kernels) and Optimized Stable Diffusion Attention kernel (fused_sd_attention_small_head). In addition to NKI kernel samples for average_pool2d, rmsnorm, tensor_addition, layernorm, transpose_2d, and matrix_multiplication.

For more information see NKI section and check the NKI samples Github repository: aws-neuron/nki-samples

NxD Training (NxDT): NxDT is a PyTorch-based library that adds support for user-friendly distributed training experience through a YAML configuration file compatible with NeMo,, allowing users to easily set up their training workflows. At the same time, NxDT maintains flexibility, enabling users to choose between using the YAML configuration file, PyTorch Lightning Trainer, or writing their own custom training script using the NxD Core. The library supports PyTorch model classes including Hugging Face and Megatron-LM. Additionally, it leverages NeMo’s data engineering and data science modules enabling end-to-end training workflows on NxDT, and providing compatability with NeMo through minimal changes to the YAML configuration file for models that are already supported in NxDT. Furthermore, the functionality of the Neuron NeMo Megatron (NNM) library is now part of NxDT, ensuring a smooth migration path from NNM to NxDT.

For more information see NxD Training (beta) and check the NxD Training Github repository: aws-neuron/neuronx-distributed-training

Training Highlights: This release adds support for Llama 3.1 8B and 70B model training up to 32K sequence length (beta). It also adds support for torch.autocast() for native PyTorch mixed precision support and PEFT LoRA model training.

Inference Highlights: Neuron 2.20 adds support for Llama 3.1 models (405b, 70b, and 8b variants) and introduces new features like on-device top-p sampling for improved performance, support for up to 128K context length through Flash Decoding, and multi-node inference for large models like Llama-3.1-405B. Furthermore, this release improves model loading in Transformers Neuronx for models like Llama-3 by loading the pre-sharded or pre-transformed weights and adds support to Diffusion-Transformers (DiT) models such as Pixart-alpha and Pixart-sigma.

Compiler: This release introduces Neuron Compiler support for RMSNorm and RMSNormDx operators, along with enhanced performance for the sort operator.

System Tools: As for the Neuron Tools, it enables NKI profiling support in the Neuron Profiler and introduces improvements to the Neuron Profiler UI.

Neuron Driver: This release adds support for the Rocky Linux 9.0 operating system.

Neuron Containers: This release introduces Neuron Helm Chart, which helps streamline the deployment of AWS Neuron components on Amazon EKS. See Neuron Helm Chart Github repository: aws-neuron/neuron-helm-charts. Additionaly, this release adds ECS support for the “Neuron Node Problem Detector and Recovery” artifact. See Neuron Problem Detector And Recovery.

Neuron DLAMIs and DLCs: This release includes the addition of the NxDT package to various Neuron DLAMIs (Multi-Framework Neuron DLAMI, PyTorch 1.13 Neuron DLAMI, and PyTorch 2.1 Neuron DLAMI) and the inclusion of NxDT in the PyTorch 1.13 Training Neuron DLC and PyTorch 2.1 Training Neuron DLC.

Software Maintenance Policy: This release also updates Neuron SDK software maintenance poclicy, For more information see Neuron Software Maintenance policy

More release content can be found in the table below and each component release notes.

2.20.0 Known Issues and Limitations#

• Known issues when using on_device_generation flag in Transformers NeuronX config for Llama models. Customers are advised not to use the flag when they see an issue. See more at Transformers Neuron (transformers-neuronx) release notes
• See component release notes below for any additional known issues.

Neuron Components Release Notes#

Inf1, Trn1/Trn1n and Inf2 common packages#

Trn1/Trn1n and Inf2 only packages#

Inf1 only packages#

Neuron 2.19.1 (07/19/2024)#

This release (Neuron 2.19.1) addresses an issue with the Neuron Persistent Cache that was introduced in the previous release, Neuron 2.19. The issue resulted in a cache-miss scenario when attempting to load a previously compiled Neuron Executable File Format (NEFF) from a different path or Python environment than the one used for the initial Neuron SDK installation and NEFF compilation. This release resolves the cache-miss problem, ensuring that NEFFs can be loaded correctly regardless of the path or Python environment used to install the Neuron SDK, as long as they were compiled using the same Neuron SDK version.

Neuron 2.19.0 (07/03/2024)#

Table of contents

• What’s New
• 2.19.0 Known Issues and Limitations
• Neuron Components Release Notes
  • Inf1, Trn1/Trn1n and Inf2 common packages
  • Trn1/Trn1n and Inf2 only packages
  • Inf1 only packages

What’s New#

Neuron 2.19 release adds Llama 3 training support and introduces Flash Attention kernel support to enable LLM training and inference for large sequence lengths. Neuron 2.19 also introduces new features and performance improvements to LLM training, improves LLM inference performance for Llama 3 model by upto 20%, and adds tools for monitoring, problem detection and recovery in Kubernetes (EKS) environments, improving efficiency and reliability.

Training highlights: LLM model training user experience using NeuronX Distributed (NxD) is improved by support for Flash Attention to enable training with longer sequence lengths >= 8K. Neuron 2.19 adds support for Llama 3 model training. This release also adds support for Interleaved pipeline parallelism to reduce idle time (bubble size) and enhance training efficiency and resource utilization for large cluster sizes.

Inference highlights: Flash Attention kernel support in the Transformers NeuronX library enables LLM inference for context lengths of up to 32k. This release also adds [Beta] support for continuous batching with mistralai/Mistral-7B-v0.2 in Transformers NeuronX.

Tools and Neuron DLAMI/DLC highlights: This release introduces the new Neuron Node Problem Detector and Recovery plugin in EKS supported Kubernetes environments:a tool to monitor the health of Neuron instances and triggers automatic node replacement upon detecting an unrecoverable error. Neuron 2.19 introduces the new Neuron Monitor container to enable easy monitoring of Neuron metrics in Kubernetes, and adds monitoring support with Prometheus and Grafana. This release also introduces new PyTorch 2.1 and PyTorch 1.13 single framework DLAMIs for Ubuntu 22. Neuron DLAMIs and Neuron DLCs are also updated to support this release (Neuron 2.19).

More release content can be found in the table below and each component release notes.

2.19.0 Known Issues and Limitations#

• Known issues when using on_device_generation flag in Transformers NeuronX config for Llama models. Customers are advised not to use the flag when they see an issue. See more at Transformers Neuron (transformers-neuronx) release notes
• See component release notes below for any additional known issues.

Neuron Components Release Notes#

Inf1, Trn1/Trn1n and Inf2 common packages#

Trn1/Trn1n and Inf2 only packages#

Note

In next releases aws-neuronx-tools and aws-neuronx-runtime-lib will add support for Inf1.

Inf1 only packages#

Neuron 2.18.2 (04/25/2024)#

Patch release with minor Neuron Compiler bug fixes and enhancements. See more in Neuron Compiler (neuronx-cc) release notes

Neuron 2.18.1 (04/10/2024)#

Neuron 2.18.1 release introduces Continuous batching(beta) and Neuron vLLM integration(beta) support in Transformers NeuronX library that improves LLM inference throughput. This release also fixes hang issues related to Triton Inference Server as well as updating Neuron DLAMIs and DLCs with this release(2.18.1). See more in Transformers Neuron (transformers-neuronx) release notes and Neuron Compiler (neuronx-cc) release notes

Neuron 2.18.0 (04/01/2024)#

Table of contents

• What’s New
• 2.18.0 Known Issues and Limitations

What’s New#

Neuron 2.18 release introduces stable support (out of beta) for PyTorch 2.1, introduces new features and performance improvements to LLM training and inference, and updates Neuron DLAMIs and Neuron DLCs to support this release (Neuron 2.18).

Training highlights: LLM model training user experience using NeuronX Distributed (NxD) is improved by introducing asynchronous checkpointing. This release also adds support for auto partitioning pipeline parallelism in NxD and introduces Pipeline Parallelism in PyTorch Lightning Trainer (beta).

Inference highlights: Speculative Decoding support (beta) in TNx library improves LLM inference throughput and output token latency(TPOT) by up to 25% (for LLMs such as Llama-2-70B). TNx also improves weight loading performance by adding support for SafeTensor checkpoint format. Inference using Bucketing in PyTorch NeuronX and NeuronX Distributed is improved by introducing auto-bucketing feature. This release also adds a new sample for Mixtral-8x7B-v0.1 and mistralai/Mistral-7B-Instruct-v0.2 in TNx.

Neuron DLAMI and Neuron DLC support highlights: This release introduces new Multi Framework DLAMI for Ubuntu 22 that customers can use to easily get started with latest Neuron SDK on multiple frameworks that Neuron supports as well as SSM parameter support for DLAMIs to automate the retrieval of latest DLAMI ID in cloud automation flows. Support for new Neuron Training and Inference Deep Learning containers (DLCs) for PyTorch 2.1, as well as a new dedicated GitHub repository to host Neuron container dockerfiles and a public Neuron container registry to host Neuron container images.

More release content can be found in the table below and each component release notes.

2.18.0 Known Issues and Limitations#

• For PyTorch 2.1 (NeuronX), slow convergence for LLaMA-2 70B training when using Zero Redundancy Optimizer (ZeRO1) can be resolved by removing all compiler flags.
• For PyTorch 2.1 (NeuronX), torch-xla 2.1 is incompatible with the default GLibC on AL2. Users are advised to migrate to Amazon Linux 2023 , Ubuntu 22 or Ubuntu 20 Operating systems.
• See component release notes below for any additional known issues.

Neuron 2.17.0 (02/13/2024)#

What’s New#

Neuron 2.17 release improves small collective communication operators (smaller than 16MB) by up to 30%, which improves large language model (LLM) Inference performance by up to 10%. This release also includes improvements in Neuron Profiler and other minor enhancements and bug fixes.

For more detailed release notes of the new features and resolved issues, see components-rn.

To learn about the model architectures currently supported on Inf1, Inf2, Trn1 and Trn1n instances, please see model_architecture_fit.

Neuron 2.16.1 (01/18/2024)#

Patch release with compiler bug fixes, updates to Neuron Device Plugin and Neuron Kubernetes Scheduler .

Neuron 2.16.0 (12/21/2023)#

Table of contents

• What’s New
• 2.16.0 Known Issues and Limitations

What’s New#

Neuron 2.16 adds support for Llama-2-70B training and inference, upgrades to PyTorch 2.1 (beta) and adds new support for PyTorch Lightning Trainer (beta) as well as performance improvements and adding Amazon Linux 2023 support.

Training highlights: NeuronX Distributed library LLM models training performance is improved by up to 15%. LLM model training user experience is improved by introducing support of PyTorch Lightning Trainer (beta), and a new model optimizer wrapper which will minimize the amount of changes needed to partition models using NeuronX Distributed primitives.

Inference highlights: PyTorch inference now allows to dynamically swap different fine-tuned weights for an already loaded model, as well as overall improvements of LLM inference throughput and latency with Transformers NeuronX. Two new reference model samples for LLama-2-70b and Mistral-7b model inference.

User experience: This release introduces two new capabilities: A new tool, Neuron Distributed Event Tracing (NDET) which improves debuggability, and the support of profiling collective communication operators in the Neuron Profiler tool.

More release content can be found in the table below and each component release notes.

2.16.0 Known Issues and Limitations#

• We recommend running multi-node training jobs on AL2023 using Amazon EKS. Parallel Cluster currently does not support AL2023.
• There are known compiler issues impacting inference accuracy of certain model configurations of Llama-2-13b when amp = fp16 is used. If this issue is observed, amp=fp32 should be used as a work around. This issue will be addressed in future Neuron releases.
• Execution time reported in neuron-profile tool is sometimes in-accurate due to a bug in how the time is captured. The bug will be addressed in upcoming Neuron releases.
• See component release notes below for any additional known issues.

Neuron 2.15.2 (11/17/2023)#

Patch release that fixes compiler issues related to performance when training using neuronx-nemo-megatron library.

Neuron 2.15.1 (11/09/2023)#

Patch release to fix execution overhead issues in Neuron Runtime that were inadvertently introduced in 2.15 release.

Neuron 2.15.0 (10/26/2023)#

Table of contents

• What’s New

What’s New#

This release adds support for PyTorch 2.0 (Beta), increases performance for both training and inference workloads, adding ability to train models like Llama-2-70B using neuronx-distributed. With this release, we are also adding pipeline parallelism support for neuronx-distributed enabling full 3D parallelism support to easily scale training to large model sizes. Neuron 2.15 also introduces support for training resnet50, milesial/Pytorch-UNet and deepmind/vision-perceiver-conv models using torch-neuronx, as well as new sample code for flan-t5-xl model inference using neuronx-distributed, in addition to other performance optimizations, minor enhancements and bug fixes.

For more detailed release notes of the new features and resolved issues, see components-rn.

To learn about the model architectures currently supported on Inf1, Inf2, Trn1 and Trn1n instances, please see model_architecture_fit.

Neuron 2.14.1 (09/26/2023)#

This is a patch release that fixes compiler issues in certain configurations of Llama and Llama-2 model inference using transformers-neuronx.

Neuron 2.14.0 (09/15/2023)#

Table of contents

• What’s New

What’s New#

This release introduces support for Llama-2-7B model training and T5-3B model inference using neuronx-distributed. It also adds support for Llama-2-13B model training using neuronx-nemo-megatron. Neuron 2.14 also adds support for Stable Diffusion XL(Refiner and Base) model inference using torch-neuronx . This release also introduces other new features, performance optimizations, minor enhancements and bug fixes. This release introduces the following:

Note

This release deprecates --model-type=transformer-inference compiler flag. Users are highly encouraged to migrate to the --model-type=transformer compiler flag.

For more detailed release notes of the new features and resolved issues, see components-rn.

To learn about the model architectures currently supported on Inf1, Inf2, Trn1 and Trn1n instances, please see model_architecture_fit.

Neuron 2.13.2 (09/01/2023)#

This is a patch release that fixes issues in Kubernetes (K8) deployments related to Neuron Device Plugin crashes and other pod scheduling issues. This release also adds support for zero-based Neuron Device indexing in K8 deployments, see the Neuron K8 release notes for more details on the specific bug fixes.

Updating to latest Neuron Kubernetes components and Neuron Driver is highly encouraged for customers using Kubernetes.

Please follow these instructions in setup guide to upgrade to latest Neuron release.

Neuron 2.13.1 (08/29/2023)#

This release adds support for Llama 2 model training (tutorial) using neuronx-nemo-megatron library, and adds support for Llama 2 model inference using transformers-neuronx library (tutorial) .

Please follow these instructions in setup guide to upgrade to latest Neuron release.

Note

Please install transformers-neuronx from https://pip.repos.neuron.amazonaws.com to get latest features and improvements.

This release does not support LLama 2 model with Grouped-Query Attention

Neuron 2.13.0 (08/28/2023)#

Table of contents

• What’s New
• 2.13.0 Known Issues and Limitations

What’s New#

This release introduces support for GPT-NeoX 20B model training in neuronx-distributed including Zero-1 optimizer capability. It also adds support for Stable Diffusion XL and CLIP models inference in torch-neuronx. Neuron 2.13 also introduces AWS Neuron Reference for Nemo Megatron library supporting distributed training of LLMs like GPT-3 175B. This release also introduces other new features, performance optimizations, minor enhancements and bug fixes. This release introduces the following:

For more detailed release notes of the new features and resolved issues, see components-rn.

To learn about the model architectures currently supported on Inf1, Inf2, Trn1 and Trn1n instances, please see model_architecture_fit.

2.13.0 Known Issues and Limitations#

• Currently we see a NaN generated when the model implementation uses torch.dtype(float32.min) or torch.dtype(float32.max) along with XLA_USE_BF16/XLA_DOWNCAST_BF16. This is because, float32.min or float32.max gets downcasted to Inf in bf16 thereby producing a NaN. Short term fix is that we can use a small/large fp32 number instead of using float32.min/float32.max. Example, for mask creation, we can use -/+1e4 instead of min/max values. The issue will be addressed in future Neuron releases.

Neuron 2.12.2 (08/19/2023)#

Patch release to fix a jemalloc conflict for all Neuron customers that use Ubuntu 22. The previous releases shipped with a dependency on jemalloc that may lead to compilation failures in Ubuntu 22 only. Please follow these instructions in setup guide to upgrade to latest Neuron release.

Neuron 2.12.1 (08/09/2023)#

Patch release to improve reliability of Neuron Runtime when running applications on memory constrained instances. The Neuron Runtime has reduced the contiguous memory requirement for initializing the Neuron Cores associated with applications. This reduction allows bringup when only small amounts of contiguous memory remain on an instance. Please upgrade to latest Neuron release to use the latest Neuron Runtime.

Neuron 2.12.0 (07/19/2023)#

Table of contents

• What’s New
• 2.12.0 Known Issues and Limitations
  • Known Issues in Ubuntu 22 Support
  • Known issues in certain resnet models on Ubuntu 20

What’s New#

This release introduces ZeRO-1 optimizer for model training in torch-neuronx , introduces beta support for GPT-NeoX, BLOOM , Llama and Llama 2(coming soon) models in transformers-neuronx. This release also adds support for model inference serving on Triton Inference Server for Inf2 & Trn1 instances, lazy_load API and async_load API for model loading in torch-neuronx, as well as other new features, performance optimizations, minor enhancements and bug fixes. This release introduces the following:

For more detailed release notes of the new features and resolved issues, see components-rn.

To learn about the model architectures currently supported on Inf1, Inf2, Trn1 and Trn1n instances, please see model_architecture_fit.

2.12.0 Known Issues and Limitations#

Known Issues in Ubuntu 22 Support#

• Several Vision and NLP models on Ubuntu 22 are not supported due to Compilation issues. Issues will be addressed in upcoming releases.
• CustomOp feature failing with seg fault on Ubuntu 22. Issue will be addressed in upcoming releases.

Known issues in certain resnet models on Ubuntu 20#

• Known issue with support for resnet-18, resnet-34, resnet-50, resnet-101 and resnet-152 models on Ubuntu 20. Issues will be addressed in upcoming releases.

Neuron 2.11.0 (06/14/2023)#

Table of contents

• What’s New

What’s New#

This release introduces Neuron Distributed, a new python library to simplify training and inference of large models, improving usability with features like S3 model caching, standalone profiler tool, support for Ubuntu22, as well as other new features, performance optimizations, minor enhancements and bug fixes. This release introduces the following:

For more detailed release notes of the new features and resolved issues, see components-rn.

To learn about the model architectures currently supported on Inf1, Inf2, Trn1 and Trn1n instances, please see model_architecture_fit.

Neuron 2.10.0 (05/01/2023)#

Table of contents

• What’s New

What’s New#

This release introduces new features, performance optimizations, minor enhancements and bug fixes. This release introduces the following:

For more detailed release notes of the new features and resolved issues, see components-rn.

To learn about the model architectures currently supported on Inf1, Inf2, Trn1 and Trn1n instances, please see model_architecture_fit.

Neuron 2.9.1 (04/19/2023)#

Minor patch release to add support for deserialized torchscript model compilation and support for multi-node training in EKS. Fixes included in this release are critical to enable training and deploying models with Amazon Sagemaker or Amazon EKS.

Neuron 2.9.0 (03/28/2023)#

Table of contents

• What’s New

What’s New#

This release adds support for EC2 Trn1n instances, introduces new features, performance optimizations, minor enhancements and bug fixes. This release introduces the following:

For more detailed release notes of the new features and resolved issues, see components-rn.

To learn about the model architectures currently supported on Inf1, Inf2, Trn1 and Trn1n instances, please see model_architecture_fit.

Neuron 2.8.0 (02/24/2023)#

Table of contents

• What’s New

What’s New#

This release adds support for EC2 Inf2 instances, introduces initial inference support with TensorFlow 2.x Neuron (tensorflow-neuronx) on Trn1 and Inf2, and introduces minor enhancements and bug fixes.

This release introduces the following:

For more detailed release notes of the new features and resolved issues, see components-rn.

Neuron 2.7.0 (02/08/2023)#

Table of contents

• What’s New

What’s New#

This release introduces new capabilities and libraries, as well as features and tools that improves usability. This release introduces the following:

For more detailed release notes of the new features and resolved issues, see components-rn.

Neuron 2.6.0 (12/12/2022)#

This release introduces the support of PyTorch 1.12 version, and introduces PyTorch Neuron (torch-neuronx) profiling through Neuron Plugin for TensorBoard. Pytorch Neuron (torch-neuronx) users can now profile their models through the following TensorBoard views:

• Operator Framework View
• Operator HLO View
• Operator Trace View

This release introduces the support of LAMB optimizer for FP32 mode, and adds support for capturing snapshots of inputs, outputs and graph HLO for debugging.

In addition, this release introduces the support of new operators and resolves issues that improve stability for Trn1 customers.

For more detailed release notes of the new features and resolved issues, see components-rn.

Neuron 2.5.0 (11/23/2022)#

Neuron 2.5.0 is a major release which introduces new features and resolves issues that improve stability for Inf1 customers.

This Neuron release is the last release that will include torch-neuron versions 1.7 and 1.8, and that will include tensorflow-neuron versions 2.5 and 2.6.

In addition, this release introduces changes to the Neuron packaging and installation instructions for Inf1 customers, see Introducing Neuron packaging and installation changes for Inf1 customers for more information.

For more detailed release notes of the new features and resolved issues, see components-rn.

Neuron 2.4.0 (10/27/2022)#

This release introduces new features and resolves issues that improve stability. The release introduces “memory utilization breakdown” feature in both Neuron Monitor and Neuron Top system tools. The release introduces support for “NeuronCore Based Sheduling” capability to the Neuron Kubernetes Scheduler and introduces new operators support in Neuron Compiler and PyTorch Neuron. This release introduces also additional eight (8) samples of models’ fine tuning using PyTorch Neuron. The new samples can be found in the AWS Neuron Samples GitHub repository.

Neuron 2.3.0 (10/10/2022)#

This Neuron 2.3.0 release extends Neuron 1.x and adds support for the new AWS Trainium powered Amazon EC2 Trn1 instances. With this release, you can now run deep learning training workloads on Trn1 instances to save training costs by up to 50% over equivalent GPU-based EC2 instances, while getting the highest training performance in AWS cloud for popular NLP models.

• Supported instances: Trn1

• Supported Frameworks: PyTorch Neuron (torch-neuronx)

• Supported Data-types

  • FP32, BF16

• Supported Rounding Modes

  • Stochastic Rounding (SR)
  • Round Nearest ties to Even (RNE)

• Supported Automatic Casting Methods

  • Neuron automatic casting of FP32 tensors / weights / operations to BF16 - Default mode
  • PyTorch automatic casting
  • Full BF16 automatic casting (via XLA_USE_BF16=1 environment variable)

• PyTorch 1.11

• Supported instances: Trn1

• Supported Python versions: Python 3.7, Python 3.8

• Eager Debug Mode

• Persistent Cache for compilation

• Collective compute operations: AllReduce

• Optimizers: AdamW, SGD

• Tested loss functions: Negative log-likelihood (NLL), Cross-entropy

• Training Libraries/Frameworks

  • torch.distributed
  • Megatron-LM Reference for Neuron
  • Training Examples
    * Multi-Layer Perceptron Training Tutorial
    * Hugging Face BERT Pretraining Tutorial (Data-Parallel)
    * PyTorch Neuron for Trainium Hugging Face BERT MRPC task finetuning using Hugging Face Trainer API
    * megatron-lm-pretraining-tutorial

• For More information:

  • PyTorch Neuron
  • Developer Guide for Training with PyTorch NeuronX
  • PyTorch Neuron (torch-neuronx) release notes
  • PyTorch Neuron (torch-neuronx) for Training Troubleshooting Guide

• Neuron Runtime 2.9

• Supported instances: Trn1, Inf1

• Elastic Fabric Adapter (EFA) @ 800Gbps

• Collective communication operators

  • AllReduce
  • AllGather
  • ReduceScatter

• Release Notes:

  • Neuron Runtime Release Notes
  • Neuron Driver Release Notes

• Neuron system tools - Adding Trn1 support to the following tools:

  • neuron-monitor
  • neuron-top
  • neuron-ls

• Release Notes:

  • Neuron System Tools

• Containers

  • Deep Learning Containers (DLC) supporting PyTorch Neuron (torch-neuronx)

• Multi-Instance distributed workloads orchestration:

  • AWS ParallelCluster (Through custom AMI build)
  • Amazon Elastic Compute Cloud (ECS)

• Supported Amazon Linux Images (AMIs)

  • Ubuntu 20 Neuron DLAMI-base (Python 3.8)
  • Amazon Linux2 Neuron DLAMI-base (Python 3.7)
  • Ubuntu 18 Neuron DLAMI-base (Python 3.7)
  • Ubuntu 18 AMI (Python 3.7)
  • Ubuntu 20 AMI (Python 3.8)
  • Amazon Linux2 AMI (Python 3.7)

The following workloads were tested in this release:

• Distributed data-parallel pre-training of Hugging Face BERT model on single Trn1.32xl instance (32 NeuronCores).
• Distributed data-parallel pre-training of Hugging Face BERT model on multiple Trn1.32xl instances.
• HuggingFace BERT MRPC task finetuning on single NeuronCore or multiple NeuronCores (data-parallel).
• Megatron-LM GPT3 (6.7B parameters) pre-training on single Trn1.32xl instance.
• Megatron-LM GPT3 (6.7B parameters) pre-training on multi Trn1.32xl instances.
• Multi-Layer Perceptron (ML) model training on single NeuronCore or multiple NeuronCores (data-parallel).
• For maximum training performance, please set environment variables XLA_USE_BF16=1 to enable full BF16 and Stochastic Rounding.

This document is relevant for: Inf1, Inf2, Trn1, Trn2