functional — Model Optimizer 0.27.1 (original) (raw)
Some supportive functions.
Classes
| ClipFunction | An universal tensor clip function. |
|---|---|
| FastHadamardTransform | The fast Hadamard transform. |
Functions
| normalized_hadamard_transform | Normalized fast hadamard transform. |
|---|
class ClipFunction
Bases: Function
An universal tensor clip function.
Pytorch’s clamp() only supports scalar range and doesn’t support broadcast. This implementation uses min/max which is more genaral. The gradient is defined according to IBM’s PACT paper https://arxiv.org/abs/1805.06085, which is also the behavior of Tensorflow’s clip_by_value()
static backward(ctx, grad_output)
Backward pass for the clip function.
static forward(ctx, input, clip_value_min, clip_value_max)
Forward pass for the clip function.
class FastHadamardTransform
Bases: Function
The fast Hadamard transform.
This only works for inputs.shape[-1] == power of 2.
static backward(ctx, grad_outputs)
Hadamard backward.
static forward(ctx, inputs)
Hadamard forward.
normalized_hadamard_transform(inputs)
Normalized fast hadamard transform.