Mean cross-entropy for a batch
Medium14 min10 hidden tests
A sorting line at a recycling plant photographs each item, and a classifier gives logits for k bins. Write mean_cross_entropy(logits, labels) that returns the training loss for a batch: the mean over items of minus the natural log of the probability the model gives the correct bin. Logits can reach 1000, so compute the loss without ever forming exp(1000). In the first example, the logits [2.0, 1.0, 0.1] with correct bin 0 give a loss of about 0.417.
Pick each item’s correct bin
loss = mean over items of −ln p[correct]bottle −ln 0.659 = 0.417, can −ln 0.751 = 0.287, box −ln 0.318 = 1.146. Mean 0.617.
Mean cross-entropy 0.617.
Examples
The correct bin has the top logit, so the loss is small
mean_cross_entropy([[2.0, 1.0, 0.1]], [0])
returns 0.41703
Huge logits stay finite; an even split costs log 2
mean_cross_entropy([[1000.0, 998.0], [0.0, 0.0]], [1, 0])
returns 1.41004
Requirements
logitsis a 2-D list or array of shape (n, k), andlabelsa list of n integers in 0 to k - 1.- The probabilities are the softmax of each row. Use the natural log.
- Return one float: the mean over the n items of
-log p[i, labels[i]]. - Stay finite for logits of 1000 and above, or -1000 and below.
The idea behind it
Cross-entropy is the standard loss for classification. For one example with logits z and correct class y, the model's probability is the softmax entry p_y, and the loss is -log p_y. A confident right answer costs almost 0. A uniform guess over k classes costs log k. A confident wrong answer costs a lot, which pushes training hard on its worst mistakes. Written with logits, the loss is log(sum_j exp(z_j)) - z_y. The first term is called log sum exp. Computing it as m + log(sum exp(z - m)), with m the row max, keeps every exponent at 0 or below, so nothing overflows, and the log never sees a 0. PyTorch's CrossEntropyLoss takes raw logits for this reason, and it averages over the batch by default so the loss scale does not depend on batch size.