Ridge regression loss

Background

Ridge regression is linear regression with an L2 penalty on the weights. The penalty discourages large coefficients, which reduces variance and stabilises the fit when features are correlated or the normal-equation system is near-singular. The ridge loss is the objective you minimise: mean squared error plus $\alpha$ times the squared L2 norm of the weights.

Problem statement

Implement ridge_loss(X, w, y_true, alpha) that returns the ridge-regression objective:

$$L = \frac{1}{n}\sum_{i=1}^{n}\big(y_i - (Xw)_i\big)^2 \; + \; \alpha \sum_{j} w_j^2$$

Input

• X — np.ndarray of shape (n, d): the design matrix.
• w — np.ndarray of shape (d,): the weight vector.
• y_true — np.ndarray of shape (n,): the target values.
• alpha — float: the L2 regularisation strength.

Output

Returns a float: the mean-squared-error term plus the L2 penalty.

Examples

Example 1

Input:  X = [[1, 1], [2, 1], [3, 1], [4, 1]], w = [0.2, 2], y_true = [2, 3, 4, 5], alpha = 0.1
Output: 2.204

Explanation: predictions $Xw = [2.2, 2.4, 2.6, 2.8]$ give residuals $[-0.2, 0.6, 1.4, 2.2]$, whose mean square is $1.8$. The penalty is $0.1\,(0.2^2 + 2^2) = 0.404$, so the loss is $1.8 + 0.404 = 2.204$.

Constraints

• Use the mean squared error (divide by $n$), then add $\alpha\lVert w\rVert_2^2$ — the penalty is not averaged and has no factor of $\tfrac12$.
• Return a single float.
• Vectorise with numpy; tests compare with atol=1e-6.

Notes

• The penalty $\alpha\sum_j w_j^2$ is the only difference from plain MSE: as $\alpha \to 0$ ridge becomes ordinary least squares, and as $\alpha \to \infty$ the weights are driven toward zero.
• Conventionally the intercept is not penalised; here every entry of w is, so only fold a bias column into X/w if you intend it to be regularised too.