Ridge regression in closed form
Medium15 min10 hidden tests
An energy team predicts a building's daily power use from a few sensor readings. Some readings move together, so plain least squares gives wild weights. Ridge regression adds a penalty on the size of the weights. Write ridge_fit(X, y, alpha) that returns the weights w and intercept b that minimize sum((y - X @ w - b) ** 2) + alpha * sum(w ** 2). The intercept is not penalized.
Raise the penalty
w = Σ(x − x̄)(y − ȳ) / (Σ(x − x̄)² + α), b = ȳ − w x̄w = 4 / (2 + 2.0) = 1, b = 4 − 1 × 2 = 2. The penalty shrinks the weight; the intercept is not penalized, so the line still passes through the means (2, 4).
Weight 1, intercept 2.
Examples
With no penalty, ridge is least squares: w = 2, b = 0
ridge_fit([[1.0], [2.0], [3.0]], [2.0, 4.0, 6.0], 0.0)
returns ([2], 0)
The penalty halves the weight, and the intercept moves to keep the line through the means
ridge_fit([[1.0], [2.0], [3.0]], [2.0, 4.0, 6.0], 2.0)
returns ([1], 2)
Requirements
Xis an (n, d) array or list of lists,yholds n targets, andalphais a float of 0 or more. Whenalphais 0, the columns of X are not collinear.- The loss uses the sum of squared errors over the rows (no mean), plus
alphatimes the sum of squared weights. The interceptbis left out of the penalty. - Return a tuple
(w, b):wis a 1-D float array of shape (d,), andbis a float.
The idea behind it
Ordinary least squares picks the weights that minimize the sum of squared errors. When features are correlated, many weight vectors fit almost equally well, and the solution can swing to huge values of opposite sign. Ridge regression adds alpha * sum(w ** 2) to the loss, which favors small weights. Setting the gradient to zero gives w = (X.T @ X + alpha * I)^-1 X.T @ y. Adding alpha to the diagonal also makes the matrix invertible even when the features are exactly collinear. The intercept should not be penalized. A model of power use should not be pulled toward 0 kilowatt hours just because the weights are. The standard way is to center the features and the target first. Then the best intercept for the centered data is 0, ridge solves for w alone, and b = mean(y) - mean(X) @ w puts the line back through the means. scikit-learn's Ridge does exactly this.