SELU activation

Background

SELU (Scaled Exponential Linear Unit) is a self-normalizing activation: with the right weight init (LeCun normal) it drives layer activations toward zero mean and unit variance automatically, removing the need for batch norm in fully-connected nets. It is an ELU scaled by a fixed factor with precisely chosen constants.

Problem statement

Implement selu(x):

$$\text{SELU}(x) = \lambda \begin{cases} x & x > 0 \\ \alpha\,(e^{x} - 1) & x \le 0 \end{cases}$$

with the fixed constants $\lambda \approx 1.0507$ and $\alpha \approx 1.6733$.

Input

• x — np.ndarray: input (any shape).

Output

Returns an np.ndarray of the same shape.

Examples

Example 1

Input:  x = [-1, 0, 1, 2]
Output: [-1.1113, 0.0, 1.0507, 2.1014]

Explanation: for $x>0$, SELU $=\lambda x$ (so $1\to1.0507$, $2\to2.1014$); for $x\le0$, $\lambda\alpha(e^x-1)$ (so $-1\to1.0507\cdot1.6733\cdot(e^{-1}-1)\approx-1.1113$); and $0\to0$.

Constraints

• Use the exact constants $\lambda = 1.0507009873554805$ and $\alpha = 1.6732632423543772$.
• Positive branch: $\lambda x$; non-positive branch: $\lambda\alpha(e^x - 1)$.
• Elementwise; tests compare with atol=1e-4.

Notes

• The constants are derived (Klambauer et al., 2017) so the activation is a fixed point of mean/variance propagation — the "self-normalizing" property.
• SELU pairs with LeCun-normal init and "alpha dropout"; outside that recipe its self-normalizing guarantee doesn't hold.