Find duplicated feature columns
Medium15 min11 hidden tests
A data team merged feature tables from three services and suspects that some columns are the same signal in other units, such as a temperature in Celsius and in Fahrenheit, or days used and days left in a 30-day trial. Write find_duplicate_features(X) that finds every column that is an exact linear copy of an earlier one: X[:, j] = a * X[:, i] + b on every row, for some numbers a and b with a nonzero.
Plot one column against another
copy: X[:, j] = a · X[:, i] + b on every rowColumn 1 = 1.8 × column 0 + 32 on every row: a copy. For this table the function returns [[1,0]].
Column 1 = 1.8 × column 0 + 32 on every row: a copy.
Examples
Column 1 is Celsius times 1.8 plus 32
find_duplicate_features([[0, 32, 4], [10, 50, 1], [25, 77, 3], [-5, 23, 2]])
returns [[1, 0]]
Days left is 30 minus days used, a copy with a = -1. The constant column is skipped
find_duplicate_features([[1, 29, 5], [4, 26, 5], [10, 20, 5], [30, 0, 5]])
returns [[1, 0]]
Requirements
Xis a 2-D list or array of shape (n, d) with n of 2 or more.- Return a list of
[j, i]pairs sorted by j, where column j copies column i and i < j. When j copies several earlier columns, use the smallest i. Return[]when there are no copies. - a may be negative, and b may be any number.
- Skip constant columns (all values equal): they are never reported as copies or as originals.
- Allow for rounding: a true copy can be off in the last few digits, so compare with a small tolerance such as 1e-9.
The idea behind it
Two columns carry the same information when one is an affine function of the other: x_j = a x_i + b with a nonzero a. Units do this (Celsius and Fahrenheit), so do complements (days used and days left), and so do joins that bring in the same field twice.
Copies hurt linear models. The columns of X become linearly dependent, X^T X cannot be inverted, and the weights stop being unique: any split of the credit between the copies fits equally well. Regularization hides the problem, but the weights still mislead anyone reading them.
The test uses correlation. Subtracting a column's mean removes b, and scaling to length 1 removes the size of a. What is left is the same unit vector for a copy, or its negative when a is negative. So the Pearson correlation, the dot product of two centered unit columns, is exactly +1 or -1 for copies and strictly between them otherwise. A constant column has no direction once centered, so it gets its own check. Feature-selection tools often run the same check with a looser cutoff, such as 0.95, to flag near copies too.