Numerical Precision — TensorRT-LLM (original) (raw)

This document describes the different quantization recipes implemented in TensorRT-LLM and contains a support matrix for the different models.

FP32, FP16 and BF16#

The different models implemented in TensorRT-LLM work with 32-bit IEEE floating-point (FP32) numbers. When checkpoints are available, the models also support 16-bit IEEE floating-point numbers (FP16) and 16-bit Bfloat16 (BF16) as described here.

Quantization and Dequantization (Q/DQ)#

Given a floating-point number x and a floating-point scaling factor s, TensorRT-LLM implements INT8 quantization as:

q = int8.satfinite(x * s)

Given an INT8 number q and a floating-point scaling factor s, TensorRT-LLM implements INT8 dequantization to the floating-point (FP) type as:

x = static_cast(q) * s

Given a matrix (2D tensor) of shape M x N (M rows and N columns) whereM is the number of tokens and N is the number of channels. TensorRT-LLM has the three following modes to quantize and dequantize the elements of the tensor:

Note that per-token and per-channel scaling modes can be used together (i.e. they are not mutually exclusive).

In pseudo-code, the quantization can be implemented as follows for the three different modes:

Per-tensor scaling.

for mi in range(M): for ni in range(N): q[mi][ni] = int8.satfinite(x[mi][ni] * s)

Per-token scaling.

for mi in range(M): for ni in range(N): q[mi][ni] = int8.satfinite(x[mi][ni] * s[mi])

Per-channel scaling.

for mi in range(M): for ni in range(N): q[mi][ni] = int8.satfinite(x[mi][ni] * s[ni])

INT8 SmoothQuant (W8A8)#

The SmoothQuant technique was introduced inhttps://arxiv.org/abs/2211.10438. It is a method to run inference using INT8 for both activations and weights while maintaining the accuracy of the network (on downstream tasks).

As explained in the research paper, preprocessing must be applied to the weights of the model. TensorRT-LLM includes scripts to prepare the model to run using the SmoothQuant method.

Examples of how to enable SmoothQuant for GPT, GPT-J and LLaMA can be found in the examples/quantization folder of that release.

INT4 and INT8 Weight-Only (W4A16 and W8A16)#

The INT4 and INT8 Weight-Only techniques consist in quantizing the weights of a model and dequantizing those weights on-the-fly in linear layers (Matmuls). The activations are encoded using floating-point values (FP16 or BF16).

To use INT4/INT8 Weight-Only methods, the user must determine the scaling factors to use to quantize and dequantize the weights of the model.

This release includes examples for GPT andLLaMA.

GPTQ and AWQ (W4A16)#

The GPTQ and AWQ techniques are presented inhttps://arxiv.org/abs/2210.17323andhttps://arxiv.org/abs/2306.00978, respectively. TensorRT-LLM supports per-group scaling factors and zero-offsetting in linear layers to implement GPTQ and AWQ methods. See theWeightOnlyGroupwiseQuantMatmulPluginplugin and the correspondingweight_only_groupwise_quant_matmulPython function, for details.

This release includes examples of applying GPTQ to GPT-NeoXand LLaMA-v2, as well as an example of using AWQ withGPT-J. Those examples are experimental implementations and are likely to evolve in a future release.

FP8 (Hopper)#

This release of TensorRT-LLM contains implementations of FP8 for GPT-NeMo, GPT-J and LLaMA. Those examples can be found inexamples/quantization.

NVFP4 (Blackwell)#

LLama and Mixtral can run in NVFP4 datatype. Those examples can be found in Llama examples.

Support matrix#

This release of TensorRT-LLM contains the following examples:

Note: The vision component of multi-modal models(BLIP2-OPT/BLIP2-T5/LLaVA/VILA/Nougat) uses FP16 by default. The language component decides which quantization methods are supported by a given multi-modal model.

Technical Detail: The QuantMode Flags#

The quantization method is controlled by theQuantMode flags. The different fields are:

There are three additional flags to control TensorRT-LLM: