Sigmoid forward + backward

Background

The sigmoid $\sigma(x) = 1/(1 + e^{-x})$ squashes any real number into $(0, 1)$, which makes it the activation of choice when you need a probability. It predates the deep-learning era and is still everywhere in binary classification heads and gates. Implementing it well means two things: a forward pass that stays finite at extreme inputs, and a backward pass that exploits the elegant $s(1-s)$ derivative.

Problem statement

Implement two functions:

$$\sigma(x) = \frac{1}{1 + e^{-x}}, \qquad\qquad \frac{\partial L}{\partial x} = \frac{\partial L}{\partial y}\cdot s\,(1 - s) \quad\text{where } s = \sigma(x)$$

sigmoid(x) must be numerically stable (no NaN/inf even at $x = \pm 1000$). sigmoid_backward(grad_out, sigmoid_out) takes the saved output $s$ (not the input) and returns the input gradient via $s(1-s)$.

Input

• sigmoid(x) — x: np.ndarray of any shape.
• sigmoid_backward(grad_out, sigmoid_out) — grad_out: upstream gradient w.r.t. the output; sigmoid_out: the output $s$ of the forward pass (same shape).

Output

• sigmoid returns an array of the same shape with values in $(0, 1)$.
• sigmoid_backward returns the gradient w.r.t. the input, same shape.

Examples

Example 1 — forward on moderate inputs

Input:  x = [-2.0, 0.0, 2.0]
Output: ≈ [0.1192, 0.5, 0.8808]

Explanation: $\sigma(0) = 0.5$ exactly, and the function is symmetric: $\sigma(-2) = 1 - \sigma(2) \approx 0.1192$.

Example 2 — backward, including the saturated tails

Input:  x = [-100, 0, 100]  ->  s = sigmoid(x) ≈ [0, 0.5, 1]
        grad_out = [1, 1, 1]
Output: ≈ [0.0, 0.25, 0.0]

Explanation: the gradient is $s(1-s)$. At $x=0$, $s=0.5$ gives the maximum $0.5\cdot0.5 = 0.25$; at the saturated tails $s\approx 0$ or $1$, so $s(1-s)\approx 0$ — the vanishing gradient.

Constraints

• Numerical stability: the naive 1/(1+exp(-x)) overflows for very negative $x$ (since exp(-x)→inf). Branch on the sign so the exp argument is always $\le 0$ — use exp(x)/(1+exp(x)) for $x < 0$, the original for $x \ge 0$. Both equal $\sigma$; only one stays finite.
• Output values lie in $(0, 1)$; at $x=\pm1000$ the result saturates to $\approx 0$ / $\approx 1$ with no NaN/inf.
• sigmoid_backward consumes the output $s$, not the input: $\dfrac{\partial L}{\partial x} = \text{grad\_out}\cdot s(1-s)$.
• The analytic backward must match a finite-difference gradient (atol≈1e-4); $\sigma'(0) = 0.25$ exactly.

Notes

• Why pass the output. Saving $s$ makes the backward a single multiply with no extra exp calls — which is why framework activation APIs hand the backward the cached output.
• Saturation = vanishing gradient. $s(1-s)\to 0$ at the tails is exactly the failure mode that kills learning in deep sigmoid stacks, and the reason ReLU (see ReLU forward + backward) was adopted instead.