Elastic-Net regression (gradient descent)

Background

Elastic Net is linear regression with both an L1 (Lasso) and an L2 (Ridge) penalty on the weights. L1 drives some coefficients exactly to zero (feature selection); L2 shrinks them smoothly and stabilises the fit when features are correlated. Elastic Net blends the two, which makes it the default when you have many, possibly-correlated features. Because the L1 term is non-differentiable at zero there is no closed form, so it is trained by gradient descent (using the L1 subgradient).

Problem statement

Implement elastic_net_gradient_descent(X, y, alpha1, alpha2, learning_rate, max_iter, tol) that fits weights and bias by gradient descent on the elastic-net objective. Initialise $w = 0$, $b = 0$ and repeat:

$$e = (Xw + b) - y$$ $$\nabla_w = \frac{1}{n} X^\top e + \alpha_1 \operatorname{sign}(w) + 2\alpha_2 w, \qquad \nabla_b = \frac{1}{n}\sum_i e_i$$ $$w \leftarrow w - \eta\,\nabla_w, \qquad b \leftarrow b - \eta\,\nabla_b$$

Stop early once $\lVert \nabla_w \rVert_1 < \text{tol}$.

Input

• X — np.ndarray of shape (n_samples, n_features): feature matrix (bias is a separate scalar, not a column).
• y — np.ndarray of shape (n_samples,): targets.
• alpha1 — float: L1 (Lasso) strength.
• alpha2 — float: L2 (Ridge) strength.
• learning_rate — float: step size $\eta$.
• max_iter — int: maximum number of gradient steps.
• tol — float: stop when the L1 norm of the weight gradient falls below this.

Output

Returns a tuple (weights, bias):

• weights — np.ndarray of shape (n_features,).
• bias — float.

Examples

Example 1

Input:  X = [[0, 0], [1, 1], [2, 2]], y = [0, 1, 2]
        alpha1 = 0.1, alpha2 = 0.1, learning_rate = 0.01, max_iter = 1000, tol = 1e-4
Output: weights = [0.3732, 0.3732], bias = 0.2479

Explanation: the data follows $y = x$ exactly, but the L1 + L2 penalties shrink each weight well below $1$ and push the leftover signal into the bias — trading a little fit for smaller, more stable coefficients.

Constraints

• Initialise weights to zeros and bias to 0.
• Gradient = data term $\frac{1}{n}X^\top e$ + L1 subgradient $\alpha_1\operatorname{sign}(w)$ + L2 term $2\alpha_2 w$.
• Stop early if $\lVert\nabla_w\rVert_1 < \text{tol}$; otherwise run max_iter steps.
• Tests compare with atol=1e-2.

Notes

• The L1 subgradient np.sign(w) is $0$ at exactly $w = 0$, so a weight that reaches zero feels no further L1 push and can stay sparse — this is what gives Lasso/Elastic-Net their feature-selection behaviour.
• Pure Lasso is $\alpha_2 = 0$; pure Ridge is $\alpha_1 = 0$; Elastic Net interpolates. The factor $2$ on the L2 term is just the derivative of $\alpha_2\lVert w\rVert_2^2$.