Float8DynamicActivationFloat8WeightConfig — torchao main documentation (original) (raw)

class torchao.quantization.Float8DynamicActivationFloat8WeightConfig(activation_dtype: dtype = torch.float8_e4m3fn, weight_dtype: dtype = torch.float8_e4m3fn, granularity: Optional[Union[PerTensor, PerRow, List[Union[PerTensor, PerRow]]]] = None, mm_config: Optional[Float8MMConfig] = None, set_inductor_config: bool = True)[source]¶

Configuration for applying float8 dynamic symmetric quantization to both activations and weights of linear layers.

Parameters:

• activation_dtype (torch.dtype) – The target data type for activation quantization. Default is torch.float8_e4m3fn.
• weight_dtype (torch.dtype) – The target data type for weight quantization. Default is torch.float8_e4m3fn.
• granularity – The granularity for quantization. Can be either a single granularity (applied to both activations and weights) or a tuple of two granularities (one for activations, one for weights). If None, defaults to PerTensor for both. Currently both quantizations need to be the same type. And only PerTensor and PerRow are supported.
• mm_config (Float8MMConfig) – Configuration for the matrix multiplication. Default uses fast accumulation.
• set_inductor_config (bool) – if True, adjusts torchinductor settings to recommended values.