torch.nn.functional.gelu — PyTorch 2.7 documentation (original) (raw)

torch.nn.functional.gelu(input, approximate='none') → Tensor¶

When the approximate argument is ‘none’, it applies element-wise the functionGELU(x)=x∗Φ(x)\text{GELU}(x) = x * \Phi(x)

where Φ(x)\Phi(x) is the Cumulative Distribution Function for Gaussian Distribution.

When the approximate argument is ‘tanh’, Gelu is estimated with

GELU(x)=0.5∗x∗(1+Tanh(2/π∗(x+0.044715∗x3)))\text{GELU}(x) = 0.5 * x * (1 + \text{Tanh}(\sqrt{2 / \pi} * (x + 0.044715 * x^3)))

See Gaussian Error Linear Units (GELUs).

• torch.nn.functional.gelu
  • gelu()

To analyze traffic and optimize your experience, we serve cookies on this site. By clicking or navigating, you agree to allow our usage of cookies. As the current maintainers of this site, Facebook’s Cookies Policy applies. Learn more, including about available controls: Cookies Policy.