Performance — NVIDIA NeMo Framework User Guide (original) (raw)
The NVIDIA NeMo Framework accelerates the entire AI workflow end-to-end, from data preparation to model training to inference. It provides optimal performance for training advanced generative AI models by incorporating the most recent training techniques, such as model parallelization, optimized attention mechanisms, and more, to achieve high training throughput. For inference, the NeMo Framework provides a path that leverages TensorRT-LLM, a specialized library for accelerating and optimizing LLM inference on NVIDIA GPUs.
Performance Summary for Large Language Models#
Below are performance benchmarks for various large language models. These results were obtained using a version of the performance recipes available here.
- Abbreviations:
- GBS: Global Batch Size
- MBS: Micro Batch Size
- FSDP: Fully Sharded Data Parallel Size
- TP: Tensor Parallel Size
- PP: Pipeline Parallel Size
- CP: Context Parallel Size
- VP: Virtual Pipeline Parallel Size
- EP: Expert Parallel Size
- GA: Number of Gradient Accumulations
Pre-training#
The table below shows the pre-training performance of various models with the FP8 precision. Specifically, we use per-tensor FP8 quantization using the scaling factors calculated in the current step (for both pre-training and fine-tuning).
- Container: NeMo 25.04
- System: DGX-GB200
- System: DGX-B200
- System: DGX-H100
Fine-Tuning#
The table below presents the fine-tuning performance of Llama3 models using Supervised Fine-Tuning (SFT) and Low-Rank Adaptors (LoRA) at FP8 precision (using NeMo 2.0).
- Container: NeMo 25.04
For fine-tuning, we use the SQuAD-v1.1 dataset, with inputs packed to 4096 tokens.
- System: DGX-GB200
- System: DGX-B200
- System: DGX-H100