Entropy and cross-entropy in nats
Medium12 min11 hidden tests
A team trains an image classifier on soft labels: for each image, several annotators vote, and the share of votes per class forms a true distribution p. The model outputs its own distribution q over the same classes. Write entropy_and_cross_entropy(p, q) that returns the entropy of p and the cross-entropy of q against p, both with the natural log. Classes with no votes contribute nothing, and a model that gives zero chance to a voted class pays an infinite cost.
Drag the model’s bars
H(p) = −Σ p ln p, H(p, q) = −Σ p ln qEntropy H(p) = 1.04 nats. Cross-entropy H(p, q) = 1.213. The model pays 0.173 nats more than the votes’ own entropy.
Entropy 1.04, cross-entropy 1.213.
Examples
When the model matches p, cross-entropy equals entropy: ln 2 = 0.693
entropy_and_cross_entropy([0.5, 0.5], [0.5, 0.5])
returns (0.693147, 0.693147)
A model that swaps two classes pays 1.21 nats against an entropy of 1.04
entropy_and_cross_entropy([0.5, 0.25, 0.25], [0.25, 0.5, 0.25])
returns (1.03972, 1.21301)
Requirements
pandqare lists or 1-D arrays of the same length k, with nonnegative entries that each sum to 1.- Entropy is H(p) = -sum of p_i ln(p_i). Cross-entropy is H(p, q) = -sum of p_i ln(q_i). Use the natural log, so both are in nats.
- Terms with p_i = 0 count as 0 in both sums.
- If some class has p_i > 0 and q_i = 0, the cross-entropy is
math.inf. - Return a tuple
(entropy, cross_entropy)of two floats.
The idea behind it
Entropy measures how uncertain a distribution is. For p over k classes,
H(p) = -sum p_i ln p_i
It is 0 when one class is certain and largest, ln k, when every class is equally likely. With the natural log the unit is the nat; with log base 2 it is the bit, and 1 nat is about 1.44 bits.
Cross-entropy scores a model's distribution q against the true p:
H(p, q) = -sum p_i ln q_i
It is the average surprise of the model when outcomes really follow p. It is never below H(p), and it equals H(p) only when q = p. The gap is the KL divergence, the topic of the next problem.
With a one-hot p, a single true class c, cross-entropy is just -ln q_c. That is the classification loss in PyTorch and Keras and the next-token loss of every language model. A model that puts q_c = 0 on a true class has infinite loss, which is why real code clips probabilities or works with logits.