Focal loss (multiclass)

Background

Focal loss (Lin et al., 2017, for RetinaNet object detection) solves a class-imbalance problem: when 99% of candidate boxes are "easy negatives" (obviously background), their many small losses swamp the gradient and drown out the rare hard examples. The fix is to down-weight examples the model already gets right, via a modulating factor on top of cross-entropy, so training keeps focusing on what it hasn't learned yet.

Problem statement

Implement focal_loss(logits, target, gamma=2.0, alpha=1.0) for one example. Let $p_t = \text{softmax}(z)_{\text{target}}$ be the predicted probability of the true class. Then:

$$L_{\text{focal}} = -\,\alpha\,(1 - p_t)^{\gamma}\,\log p_t$$

The factor $(1-p_t)^\gamma$ does the work: confident-correct ($p_t\!\to\!1$) examples have it $\to 0$ (loss crushed), while hard ($p_t$ small) examples keep it $\approx 1$ (loss ≈ cross-entropy). Compute the softmax stably and clip $p_t$ away from 0.

Input

• logits — 1-D np.ndarray of shape (K,): the pre-softmax scores for one example.
• target — int: the true class index in $[0, K)$.
• gamma — float $\ge 0$: the focusing parameter ($\gamma=0$ reduces to weighted CE; $\gamma=2$ is the RetinaNet default).
• alpha — float $\ge 0$: the class-balancing weight.

Output

Returns a float: the scalar focal loss.

Examples

Example 1 — $\gamma = 0$ reduces to (weighted) cross-entropy

Input:  logits = [1.0, 2.0, 3.0, 0.5], target = 2, gamma = 0, alpha = 1
Output: -log(p_t)        # plain cross-entropy

Explanation: with $\gamma=0$ the modulating factor $(1-p_t)^0 = 1$, so the loss collapses to $-\alpha\log p_t$ — exactly $\alpha$-weighted cross-entropy.

Example 2 — the modulation factor scales the loss

Input:  logits = [0.0, 0.0, 0.0], target = 1, gamma = 2, alpha = 1
Output: ≈ 0.488

Explanation: uniform logits give $p_t = 1/3$, so the factor is $(1 - 1/3)^2 = (2/3)^2 \approx 0.444$. The focal loss is that fraction of the plain CE $-\log(1/3)\approx1.0986$, i.e. $0.444\times1.0986\approx0.488$.

Constraints

• Compute $p_t$ from a numerically-stable softmax (subtract logits.max()); clip $p_t$ to a tiny floor (e.g. 1e-30) to avoid log(0).
• $L = -\alpha\,(1-p_t)^\gamma\log p_t$. With $\gamma=0$ it equals $\alpha$-weighted CE.
• alpha scales the loss linearly (doubling alpha doubles the loss).
• A confident-correct prediction yields a loss far below plain CE; an uncertain one (small $p_t$) yields roughly CE. Stable for large logits (e.g. [1000, 1001, 1002]).

Notes

• Three regimes of $(1-p_t)^\gamma$. $p_t\approx1$ → factor $\to 0$ (the model "moves on"); $p_t\approx0$ → factor $\to 1$ (full CE, reweighted by $\alpha$); $p_t\approx0.5$ → factor $\approx 0.5^\gamma$ (moderately reduced).
• Defaults & uses. $\gamma=2$, $\alpha=0.25$ (for the rare class) is the canonical RetinaNet setting; focal loss is standard in object detection (RetinaNet, YOLO derivatives) and any severely imbalanced classifier.