GELU forward (tanh approximation)

Background

GELU (Gaussian Error Linear Unit) is the smooth, probabilistic cousin of ReLU that GPT-2, BERT, and most transformer-era architectures adopted. The exact definition is $x\,\Phi(x)$, where $\Phi$ is the standard-normal CDF — but that is slow, so OpenAI introduced a tanh approximation (used by GPT-2, nanoGPT, and nn.GELU(approximate="tanh")) that matches the exact form within $10^{-4}$ and is ~3× faster.

Problem statement

Implement gelu(x) using the tanh approximation, elementwise:

$$\text{GELU}(x) \approx 0.5\,x\,\Big(1 + \tanh\!\big(\sqrt{\tfrac{2}{\pi}}\,(x + 0.044715\,x^3)\big)\Big)$$

The constants are not analytically clean — $\sqrt{2/\pi}\approx0.7978$ and $0.044715$ is exact from Hendrycks & Gimpel (2016); use them as given.

Input

• x — np.ndarray of any shape.

Output

Returns an np.ndarray of the same shape.

Examples

Example 1 — a few hand-checked points

Input:  x = [-0.5, 0.0, 2.0]
Output: ≈ [-0.154, 0.0, 1.954]

Explanation: $\text{GELU}(0) = 0$ exactly (the tanh argument is 0). $\text{GELU}(2.0)\approx1.954$ — already close to $x$. $\text{GELU}(-0.5)\approx-0.154$ — a small negative value, not zero.

Example 2 — asymptotes, and the negative leak

Input:  x = [1000, -1000]   -> ≈ [1000, 0]
        x = [-0.5]          -> ≈ -0.154        (ReLU would give 0)

Explanation: for large positive $x$, $\tanh\to 1$ so $\text{GELU}\to 0.5\,x\cdot 2 = x$; for large negative $x$, $\tanh\to -1$ so $\text{GELU}\to 0$. In between, moderately-negative inputs pass a little signal through — unlike ReLU, which hard-zeros every $x\le 0$.

Constraints

• Translate the formula directly; it is a single elementwise expression over x.
• $\text{GELU}(0) = 0$ exactly; for $x \gg 0$, $\text{GELU}(x)\to x$; for $x \ll 0$, $\text{GELU}(x)\to 0$.
• A moderate negative like $x=-0.5$ produces a small negative output ($\approx-0.154$), not 0.
• Use the constants as given — do not re-derive them.

Notes

• Why GELU over ReLU. Its gradient is continuous at 0 (no kink), which behaves better with Adam, and it lets a small fraction of negative input through, avoiding ReLU's "dying unit" problem.
• Pairs with the backward. See GELU backward for the derivative; modern refinements (SwiGLU, GeGLU) build on this, but GELU is still the default in most public transformers.