BitsandbytesPrecision — PyTorch Lightning 2.5.1.post0 documentation (original) (raw)

class lightning.pytorch.plugins.precision.BitsandbytesPrecision(mode, dtype=None, ignore_modules=None)[source]¶

Bases: Precision, BitsandbytesPrecision

Plugin for quantizing weights with bitsandbytes.

Note

The optimizer is not automatically replaced with bitsandbytes.optim.Adam8bit or equivalent 8-bit optimizers.

Parameters:

• mode¶ (Literal['nf4', 'nf4-dq', 'fp4', 'fp4-dq', 'int8', 'int8-training']) – The quantization mode to use.
• dtype¶ (Optional[dtype]) – The compute dtype to use.
• ignore_modules¶ (Optional[set[str]]) – The submodules whose Linear layers should not be replaced, for example. {"lm_head"}. This might be desirable for numerical stability. The string will be checked in as a prefix, so a value like “transformer.blocks” will ignore all linear layers in all of the transformer blocks.