Leaky ReLU activation

Background

Leaky ReLU fixes ReLU's "dying neuron" problem by letting a small gradient flow for negative inputs: instead of zeroing them, it scales them by a small slope $\alpha$. This keeps every unit slightly active so it can recover during training.

Problem statement

Implement leaky_relu(x, alpha=0.01):

$$\text{LeakyReLU}(x) = \begin{cases} x & x > 0 \\ \alpha x & x \le 0 \end{cases}$$

Input

• x — np.ndarray: input (any shape).
• alpha — float: the negative slope (default 0.01).

Output

Returns an np.ndarray of the same shape.

Examples

Example 1

Input:  x = [-2, -0.5, 0, 3], alpha = 0.1
Output: [-0.2, -0.05, 0.0, 3.0]

Explanation: positive inputs pass through unchanged; negatives are scaled by $\alpha=0.1$, so $-2\to-0.2$ and $-0.5\to-0.05$.

Constraints

• Positive (and zero) inputs are unchanged; negatives are multiplied by alpha.
• Elementwise; preserves shape.
• $\alpha=0$ recovers plain ReLU.

Notes

• Unlike ReLU, the negative slope keeps gradients non-zero everywhere, so a unit pushed negative is never permanently dead.
• PReLU makes $\alpha$ a learnable per-channel parameter; Leaky ReLU keeps it fixed (commonly 0.01).