Logistic regression — gradient descent

Background

Logistic regression is the workhorse binary classifier: it passes a linear score $X\theta$ through the sigmoid $\sigma(z) = 1/(1+e^{-z})$ to produce a probability, then is trained by minimising binary cross-entropy (log loss). Unlike linear regression's normal equation there is no closed form — the parameters are found by gradient descent. It underpins everything from click-through prediction to the final layer of a neural-network classifier.

Problem statement

Implement train_logreg(X, y, learning_rate, iterations) that trains logistic regression by gradient descent on the binary cross-entropy loss and returns the learned coefficients together with the loss recorded at every iteration. Prepend a bias column of ones to $X$, start from $\theta = 0$, and at each step use the sigmoid prediction and the BCE gradient:

$$\hat{y} = \sigma(X\theta), \qquad \sigma(z) = \frac{1}{1 + e^{-z}}$$ $$\theta \leftarrow \theta - \alpha\, X^\top (\hat{y} - y)$$ $$L = -\sum_{i} \left[\, y_i \log \hat{y}_i + (1 - y_i)\log(1 - \hat{y}_i) \,\right]$$

Input

• X — np.ndarray of shape (n_samples, n_features): the feature matrix (no bias column — the function prepends one).
• y — np.ndarray of shape (n_samples,): binary labels in $\{0, 1\}$.
• learning_rate — float: the gradient-descent step size $\alpha$.
• iterations — int: the number of full-batch gradient steps.

Output

Returns a tuple (coefficients, losses):

• coefficients — list[float] of length n_features + 1 (bias first), each rounded to 4 decimals.
• losses — list[float] of length iterations: the total BCE loss after each step, rounded to 4 decimals.

Examples

Example 1

Input:  X = [[1.0, 0.5], [-0.5, -1.5], [2.0, 1.5], [-2.0, -1.0]]
        y = [1, 0, 1, 0], learning_rate = 0.01, iterations = 20
Output: coefficients = [-0.0003, 0.4038, 0.3379]
        losses[0] = 2.7726, falling monotonically over the 20 steps

Explanation: with $\theta = 0$ every prediction is $\sigma(0) = 0.5$, so the first loss is $-4\log(0.5) = 4\ln 2 \approx 2.7726$. Each gradient step raises $\hat{y}$ on the positives and lowers it on the negatives, so the bias stays near $0$ while the two feature weights grow to separate the classes and the loss falls monotonically.

Constraints

• Prepend a single bias column of ones to X; initialise $\theta$ to zeros.
• Use the summed (not averaged) BCE loss, with gradient $X^\top(\hat{y} - y)$.
• Round the coefficients and every loss value to 4 decimal places.
• The learning_rate is small enough that the loss is non-increasing; tests compare with atol=1e-3.

Notes

• The BCE gradient simplifies beautifully: despite the $\log\sigma$ in the loss, $\partial L / \partial\theta = X^\top(\hat{y} - y)$ — the same clean form as linear regression's gradient, which is why logistic regression trains so stably.
• Numerical care: $\log\hat{y}$ diverges if $\hat{y}$ reaches exactly $0$ or $1$. With zero init and few iterations this won't happen, but production code clips $\hat{y}$ into $[\varepsilon,\, 1-\varepsilon]$.