Fit a line by gradient descent
Medium14 min11 hidden tests
A courier team wants a quick rule for delivery time: minutes equal slope times kilometers plus a fixed intercept. Write fit_line(x, y, lr, steps) that fits the slope and intercept by full-batch gradient descent on the mean squared error, mean((slope * x + intercept - y) ** 2). Both start at 0, and each step updates them together from the gradients at the current values.
Run more steps
loss = mean((slope · x + intercept − y)²)From 0, d/dslope = mean(2(ŷ − y) x) = −22.67 and d/dintercept = mean(2(ŷ − y)) = −10.00, so slope = 0 − 0.05 × (−22.67) = 1.1333 and intercept = 0.5. The slope moves more: x scales its gradient.
Step 1: slope 1.1333, intercept 0.5.
Examples
One step from zero: the slope moves more, since x scales its gradient
fit_line([1.0, 2.0, 3.0], [3.0, 5.0, 7.0], 0.05, 1)
returns (1.13333, 0.5)
After enough steps the fit reaches y = 2x + 1
fit_line([1.0, 2.0, 3.0], [3.0, 5.0, 7.0], 0.05, 2000)
returns (2, 1)
Requirements
xandyare lists or 1-D arrays of the same length n,lris a positive float, andstepsis 0 or more.- The loss is the plain mean of the squared errors over all n points, with no 1/2 in front.
- Each step computes both gradients at the current slope and intercept first, then updates both.
- Return a tuple
(slope, intercept)of two floats. Withsteps = 0, return(0.0, 0.0).
The idea behind it
Fitting a line has a closed-form answer, but gradient descent on the same problem shows the training loop of every neural network in miniature. The model is pred = slope * x + intercept, and the loss is the mean squared error over all n points. With residuals r = pred - y, the chain rule gives dL/dslope = 2 * mean(r * x) and dL/dintercept = 2 * mean(r). Full-batch means each step uses all the points. The update must be simultaneous: both gradients come from the same parameters, then both parameters move. Updating one first and reusing it for the other gives a different algorithm that converges along a different path. The learning rate has a limit set by the data. Large x values make the slope's gradient large, and a learning rate that is fine for x near 1 can make the fit blow up for x near 30, which is one reason people scale their features first.