Overview — TensorRT-LLM (original) (raw)

This document summarizes performance measurements of TensorRT-LLM on a number of GPUs across a set of key models.

The data in the following tables is provided as a reference point to help users validate observed performance. It should not be considered as the peak performance that can be delivered by TensorRT-LLM.

We attempted to keep commands as simple as possible to ease reproducibility and left many options at their default settings. Tuning batch sizes, parallelism configurations, and other options may lead to improved performance depending on your situaiton.

For DeepSeek R1 performance, please check out our performance guide

Throughput Measurements#

The below table shows performance data where a local inference client is fed requests at an infinite rate (no delay between messages), and shows the throughput scenario under maximum load. The reported metric is Total Output Throughput (tokens/sec).

The performance numbers below were collected using the steps described in this document.

Testing was performed on models with weights quantized using ModelOpt and published by NVIDIA on the Model Optimizer HuggingFace Collection.

FP4 Models:#

nvidia/Llama-3.3-70B-Instruct-FP4 nvidia/Llama-3.1-405B-Instruct-FP4

Llama 3.3 70B FP4#

Llama 3.1 405B FP4#

FP8 Models:#

nvidia/Llama-3.1-8B-Instruct-FP8 nvidia/Llama-3.1-70B-Instruct-FP8 nvidia/Llama-3.1-405B-Instruct-FP8

Llama 3.1 8B FP8#

Llama 3.1 70B FP8#

Llama 3.1 405B FP8#

Reproducing Benchmarked Results#

[!NOTE] The only models supported in this workflow are those listed in the table above.

The following tables are references for commands that are used as part of the benchmarking process. For a more detailed description of this benchmarking workflow, see the benchmarking suite documentation.

Command Overview#

Starting with v0.19, testing was performed using the PyTorch backend - this workflow does not require an engine to be built.

Variables#

Preparing a Dataset#

In order to prepare a dataset, you can use the provided script. To generate a synthetic dataset, run the following command:

python benchmarks/cpp/prepare_dataset.py --tokenizer=$model_name --stdout token-norm-dist --num-requests=$num_requests --input-mean=$isl --output-mean=$osl --input-stdev=0 --output-stdev=0 > $dataset_file

The command will generate a text file located at the path specified $dataset_file where all requests are of the same input/output sequence length combinations. The script works by using the tokenizer to retrieve the vocabulary size and randomly sample token IDs from it to create entirely random sequences. In the command above, all requests will be uniform because the standard deviations for both input and output sequences are set to 0.

For each input and output sequence length combination, the table below details the $num_requests that were used. For shorter input and output lengths, a larger number of messages were used to guarantee that the system hit a steady state because requests enter and exit the system at a much faster rate. For longer input/output sequence lengths, requests remain in the system longer and therefore require less requests to achieve steady state.

Running the Benchmark#

To run the benchmark with the generated data set, simply use the trtllm-bench throughput subcommand. The benchmarker will run an offline maximum throughput scenario such that all requests are queued in rapid succession. You simply need to provide a model name (HuggingFace reference or path to a local model), a generated dataset, and a file containing any desired extra options to the LLMApi (details in tensorrt_llm/llmapi/llm_args.py:LlmArgs).

trtllm-bench --model modelnamethroughput−−datasetmodel_name throughput --dataset modeln​amethroughput−−datasetdataset_file --backend pytorch --extra_llm_api_options $llm_options

llm_options.yml

use_cuda_graph: true cuda_graph_padding_enabled: true cuda_graph_batch_sizes:

In majority of cases, we also use a higher KV cache percentage by setting --kv_cache_free_gpu_mem_fraction 0.95 in the benchmark command. This allows us to obtain better performance than the default setting of 0.90. We fall back to 0.90 if we hit an out of memory issue.

The results will be printed to the terminal upon benchmark completion. For example,

=========================================================== = PERFORMANCE OVERVIEW

Request Throughput (req/sec): 43.2089 Total Output Throughput (tokens/sec): 5530.7382 Per User Output Throughput (tokens/sec/user): 2.0563 Per GPU Output Throughput (tokens/sec/gpu): 5530.7382 Total Token Throughput (tokens/sec): 94022.5497 Total Latency (ms): 115716.9214 Average request latency (ms): 75903.4456 Per User Output Speed [1/TPOT] (tokens/sec/user): 5.4656 Average time-to-first-token [TTFT] (ms): 52667.0339 Average time-per-output-token [TPOT] (ms): 182.9639

-- Per-Request Time-per-Output-Token [TPOT] Breakdown (ms)

[TPOT] MINIMUM: 32.8005 [TPOT] MAXIMUM: 208.4667 [TPOT] AVERAGE: 182.9639 [TPOT] P50 : 204.0463 [TPOT] P90 : 206.3863 [TPOT] P95 : 206.5064 [TPOT] P99 : 206.5821

-- Per-Request Time-to-First-Token [TTFT] Breakdown (ms)

[TTFT] MINIMUM: 3914.7621 [TTFT] MAXIMUM: 107501.2487 [TTFT] AVERAGE: 52667.0339 [TTFT] P50 : 52269.7072 [TTFT] P90 : 96583.7187 [TTFT] P95 : 101978.4566 [TTFT] P99 : 106563.4497

-- Request Latency Breakdown (ms) -----------------------

[Latency] P50 : 78509.2102 [Latency] P90 : 110804.0017 [Latency] P95 : 111302.9101 [Latency] P99 : 111618.2158 [Latency] MINIMUM: 24189.0838 [Latency] MAXIMUM: 111668.0964 [Latency] AVERAGE: 75903.4456

[!WARNING] In some cases, the benchmarker may not print anything at all. This behavior usually means that the benchmark has hit an out of memory issue. Try reducing the KV cache percentage using the --kv_cache_free_gpu_mem_fraction option to lower the percentage of used memory.