One step of logistic regression
Medium14 min10 hidden tests
A bank's loan model predicts the probability that an applicant repays with logistic regression: p = sigmoid(X @ w + b). It trains on the mean binary cross-entropy, with the natural log. Write logistic_step(X, y, w, b, lr) that computes the predictions for the whole batch at once and returns the weights and bias after one gradient descent step.
Take gradient steps
w −= lr · Xᵀ(p − y) / n, b −= lr · mean(p − y)All weights are 0, so p = 0.5 for everyone and there is no boundary yet. Press Step.
Weights at zero.
Examples
From zero weights every p is 0.5
logistic_step([[1.0, 2.0], [2.0, 1.0], [0.0, -1.0]], [1, 0, 0], [0.0, 0.0], 0.0, 0.1)
returns ([-0.0166667, 0.0333333], -0.0166667)
Confident right answers barely move the weight
logistic_step([[3.0], [-3.0]], [1, 0], [2.0], 0.5, 0.5)
returns ([2.00418], 0.499358)
Requirements
Xis an (n, d) array or list of lists,yholds n labels of 0 or 1,wholds d weights,bis a float andlris a float of 0 or more.- The loss is
-mean(y * log(p) + (1 - y) * log(1 - p))withp = 1 / (1 + exp(-(X @ w + b))). EveryX @ w + bin the tests is between -30 and 30. - Use vectorized NumPy for the whole batch, with no loop over rows.
- Return a tuple
(w_new, b_new):w_newis a 1-D float array of shape (d,), andb_newis a float.
The idea behind it
Logistic regression scores each example with z = X @ w + b and turns the score into a probability with the sigmoid, p = 1 / (1 + exp(-z)). It trains on the mean binary cross-entropy. The calculus is kind here: the derivative of the loss with respect to each score is (p - y) / n, prediction minus label, because the sigmoid's slope cancels against the log. The chain rule then gives grad_w = X.T @ (p - y) / n and grad_b = mean(p - y). That is the same shape as the gradient of mean squared error for linear regression, without the factor 2 and with probabilities in place of raw scores. Vectorizing matters: one matrix product handles every row, which is thousands of times faster than a Python loop and is how the same step runs on a GPU. A confident correct prediction has p close to y and a tiny error, so it barely moves the weights. A confident wrong one pushes hard.