Calibrate Jev's Probabilities and Thresholds

Turn Jev's probabilities into numbers you can set a policy on. Build the reuse check of a semantic cache, measure how well Jev's probabilities match reality with reliability tables, ECE and Brier score, recalibrate them with Platt scaling, pick a threshold for a precision target on a dev split and confirm it on a test split, then test whether a different question wording really ranks better.

60 min5 steps3 domainsIntermediate

Hands-on labs require Pro · $29.99/mo · cancel anytime

What you'll learn

1
Collect Jev's probabilities
Tallyboard's support bot caches every answer it writes. When a new
2
Is 0.8 really 80%?
A probability is calibrated when it matches how often things happen: of
3
Recalibrate with Platt scaling
Jev ranks the pairs well, and its numbers sit too close to the middle.
4
A threshold for a precision target
The team's rule is that a reused answer must be right at least 98% of
5
Does the wording matter?
Calibration fixes timid numbers and leaves the order of the pairs as it

Step 1, as you will see it

This is the lab’s own text. Each step ends with a check that runs your work in the lab environment; the hint and the solution stay inside the lab.

Step 1: Collect Jev's probabilities

Tallyboard's support bot caches every answer it writes. When a new question arrives, it finds the closest cached question and asks Jev one thing: can the cached answer be sent, unchanged, as the answer to the new question? A yes skips a model call and replies at once.

pairs.jsonl holds 160 such pairs, each labelled reusable by hand. Many are easy: a paraphrase ("team plan price per user?") or a different topic. The hard ones differ by one detail. "Team, 5 seats, annual" against "Team, 6 seats, annual" is not reusable. "Can we pay in euros?" against "Can we pay in British pounds?" is: the cached "No. We bill only in US dollars." answers both.

The pairs are split by cache entry. Eight cached questions and their pairs form dev; the other eight form test. Everything you tune in this lab is tuned on dev and then checked on questions it never saw.

Do this

Open calib.py.

1. Write ask(pair, question=REUSE_QUESTION, post=jev.post). One post with the state {"cached_question", "cached_answer", "new_question"} from the pair and a single noul question named "reuse" whose instructions are question. Return the probability.

2. Write collect(pairs, question=REUSE_QUESTION, post=jev.post). Ask about every pair through ThreadPoolExecutor(16), keep the order, and return a numpy array.

3. Run. A few dev pairs with their probability and label, then a histogram of the dev split by probability. Reusable pairs average about 0.66 and the rest about 0.10, and almost no pair crosses sides. Look where the reusable pairs sit, though: most of them are between 0.5 and 0.9, and few are above 0.9.

Prerequisites

Python with numpy arrays
Precision and recall, at the level of knowing what they count
No API keys: the lab's proxy provides access to Jev

Exam domains covered

CalibrationEvaluationDecision Models

Skills & technologies you'll practice

This intermediate-level ai/ml lab gives you real-world reps across:

JevTypeSafecalibrationPlatt scalingECEBrier scoresemantic cachedecision modelsintermediate

How to calibrate a decision model's probabilities

A decision model that returns probabilities invites a simple policy: act when the probability is above some number. That policy only works if the numbers mean what they say. A model can rank cases almost perfectly and still be underconfident or overconfident, and then a threshold such as 0.9 either lets mistakes through or throws most of the good cases away. In this lab you measure that for Jev, TypeSafe's decision model, on the reuse check of a semantic cache. You build reliability tables and compute expected calibration error and the Brier score. You recalibrate with Platt scaling fitted on a dev split, and choose the threshold that meets a precision target, then confirm it on a test split made of cache entries the fit never saw. Last, you compare question wordings with AUC and a paired bootstrap, and see why a small AUC lead on one sample is no reason to switch.

Frequently asked questions

Do I need a Jev or OpenRouter API key for this lab?

No. The lab sandbox reaches Jev through Preporato's model proxy, which holds the key.

What is calibration?

A model is calibrated when its probabilities match observed frequencies: of all the cases it scores 0.8, about 80% turn out true. The lab measures it with a reliability table, expected calibration error (ECE) and the Brier score.

Why not just pick the threshold on raw probabilities?

You can, and recalibrating with a monotone map such as Platt scaling does not change which cases can be separated. What it changes is what the number means: after calibration a threshold of 0.9 really means about 90% likely, so a precision target can be written as a probability and it still holds on new data.

What is Platt scaling?

A logistic regression fitted on the logit of the model's probability, sigmoid(a * logit(p) + b). An a above 1 stretches underconfident probabilities apart; b shifts them. It needs only a small labelled set, which is why the lab fits it on the dev split.