Gaussian Error Linear Units (GELUs) (original) (raw)

View PDF

Abstract:We propose the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function. The GELU activation function is xPhi(x)x\Phi(x)xPhi(x), where Phi(x)\Phi(x)Phi(x) the standard Gaussian cumulative distribution function. The GELU nonlinearity weights inputs by their value, rather than gates inputs by their sign as in ReLUs ($x\mathbf{1}_{x>0}$). We perform an empirical evaluation of the GELU nonlinearity against the ReLU and ELU activations and find performance improvements across all considered computer vision, natural language processing, and speech tasks.

Submission history

From: Dan Hendrycks [view email]
[v1] Mon, 27 Jun 2016 19:20:40 UTC (435 KB)
[v2] Fri, 8 Jul 2016 18:32:46 UTC (608 KB)
[v3] Sun, 11 Nov 2018 07:40:32 UTC (3,013 KB)
[v4] Wed, 8 Jul 2020 16:41:42 UTC (3,015 KB)
[v5] Tue, 6 Jun 2023 01:53:32 UTC (3,016 KB)