Gradient of mean squared error
Medium14 min10 hidden tests
A food delivery app predicts how many minutes late an order will be from a few trip features, with a linear model pred = X @ w + b. The training loss is the mean squared error over the n trips, L = mean((pred - y) ** 2). Write mse_gradient(X, y, w, b) that returns the gradient of L with respect to the weights w and with respect to the bias b.
Drag the point on the loss map
grad_w = 2 mean((ŷ − y) x), grad_b = 2 mean(ŷ − y)With w = 0.5 and b = 0, ŷ − y = [−2.50, −4.00, −5.50]. grad_w = 2 × mean((ŷ − y) x) = −18.000, grad_b = 2 × mean(ŷ − y) = −8.000. The arrow points against the gradient: downhill on the map.
Gradient w −18.000, b −8.000.
Examples
Both predictions are too low, so every gradient is negative
mse_gradient([[1.0, 2.0], [3.0, 1.0]], [5.0, 4.0], [0.5, 1.0], 0.0)
returns ([-7, -6.5], -4)
A perfect fit has a zero gradient
mse_gradient([[1.0], [2.0], [3.0]], [3.0, 5.0, 7.0], [2.0], 1.0)
returns ([0], 0)
Requirements
Xis an (n, d) array or list of lists,yholds n targets,wholds d weights andbis a float.- The loss is the plain mean of the squared errors, with no 1/2 in front, so a factor of 2 stays in the gradient.
- Return a tuple
(grad_w, grad_b):grad_wis a 1-D float array of shape (d,), andgrad_bis a float.
The idea behind it
Mean squared error, L = (1/n) * sum((pred_i - y_i) ** 2), is the default loss for predicting a number. For a linear model, pred = X @ w + b. Call r = pred - y the residuals. The chain rule gives dL/dpred_i = 2 * r_i / n. Each prediction depends on weight j through feature X[i, j] and on the bias with slope 1. Summing over the rows gives grad_w = (2/n) * X.T @ r and grad_b = (2/n) * sum(r). Some texts define the loss with a 1/2 in front so the 2 cancels. Both conventions are fine, and they differ by a factor that the learning rate absorbs, but the code and the loss must agree. The transpose X.T @ r is the pattern behind the backward pass of every dense layer: the gradient for a weight matrix is its input transposed times the gradient at its output.