KL divergence for drift monitoring
Medium12 min12 hidden tests
A fraud model was trained on one mix of merchant categories, and the monitoring job checks whether live traffic still looks like it. Each day it counts transactions per category, today and in the training data, and computes the KL divergence (Kullback-Leibler divergence) from today's mix to the training mix. Write kl_divergence(p_counts, q_counts) that turns both sets of counts into distributions and returns D_KL(P || Q) in nats, with the zero cases handled.
Drag today’s mix
D(P ‖ Q) = Σ P ln(P / Q)Counts become shares by dividing by each day’s total. D(P ‖ Q) = 0.11 nats. The biggest term is Gifts: 0.2 today against 0.1 in training, 0.2 × ln(0.2 / 0.1) = 0.139.
KL divergence 0.11.
Examples
Live traffic leans to the first category
kl_divergence([30, 10], [20, 20])
returns 0.130812
A category with no traffic today adds nothing
kl_divergence([5, 0, 5], [2, 1, 1])
returns 0.346574
Today has a category the training data never saw
kl_divergence([1, 1], [1, 0])
returns inf
Requirements
p_counts(today) andq_counts(training) are lists or 1-D arrays of the same length k, with nonnegative entries and a positive total each.- First divide each by its own total to get distributions P and Q.
- Return D_KL(P || Q) = sum of P_i ln(P_i / Q_i) as a float, using the natural log.
- Terms with P_i = 0 count as 0. If some P_i > 0 has Q_i = 0, return
math.inf.
The idea behind it
KL divergence measures how different a distribution P is from a reference Q:
D(P || Q) = sum P_i ln(P_i / Q_i)
It is the extra surprise, in nats, of using Q when outcomes follow P, and it equals cross-entropy H(P, Q) minus entropy H(P). It is 0 exactly when P = Q and positive otherwise. It is not symmetric: D(P || Q) and D(Q || P) usually differ, so the order is part of the definition.
The zero cases follow from that meaning. A category that never happens under P costs nothing. A category that happens under P but has zero probability under Q is infinitely surprising, so the divergence is infinite. Monitoring jobs often add a small count to every category (smoothing) so a single new category does not make the number blow up.
KL divergence shows up across ML: drift monitors compare live features with training data, variational autoencoders use it as a penalty, and RLHF (reinforcement learning from human feedback) fine-tuning keeps a model close to its starting point with a KL term.