Preporato
Find Where Jev Fails Before Your Users Do
Hosted · ide
Beta

Find Where Jev Fails Before Your Users Do

Build the probes that tell you whether a question is safe to hand to Jev, TypeSafe's decision model. Catch facts missing from the state, confident answers to arithmetic that code does exactly, closed-world choice questions that force off-topic input into a real option, and instability under reordered options or text that argues with the model, then turn the results into a ship or fix verdict per feature.

50 min5 steps3 domainsIntermediate

Hands-on labs require Pro · $29.99/mo · cancel anytime

Map the attack surface
Query
Retriever
LLM
Poisoned doc
retrieved chunk
Answer
0%
Attack-success rate
Attacks blocked · benign answers pass
graded on real output, not the model's talk

What you'll learn

  1. 1
    Facts missing from the state
    Jev answers from the state you send and nothing else. It has no clock,
  2. 2
    Confidently wrong: what code can compute
    Billing wants to ask whether an invoice adds up: every line is quantity
  3. 3
    Closed-world choices
    A choice question makes Jev pick one of your options. If an input fits
  4. 4
    Option order and text that talks to Jev
    Two more things can go wrong with any question, and both are cheap to
  5. 5
    A verdict per feature
    The probes become useful when each feature gets one clear answer that

Step 1, as you will see it

This is the lab’s own text. Each step ends with a check that runs your work in the lab environment; the hint and the solution stay inside the lab.

Step 1: Facts missing from the state

Jev answers from the state you send and nothing else. It has no clock, no database and no way to look anything up. A question whose answer depends on a fact outside the state still gets a probability back, and that probability looks just as official as a good one.

Checkout wants to ask "Has this coupon expired?" with the coupon's code and expiry date as the state. coupons.jsonl holds 40 coupons, labelled against the lab's clock, TODAY = "2026-09-24".

The probe for this failure is a baseline. If Jev is not clearly more accurate than always guessing the most common answer, it has no usable signal on the question, however confident its numbers look.

Do this

Open probes.py.

1. Write coupon_state(coupon, today=None): {"code", "expires"}, plus "today" when it is given.

2. Write signal_report(p, y). Return:

  • accuracy: at a 0.5 threshold.
  • majority: the accuracy of always guessing the more common label.
  • auc: the ranking quality, from the helper auc(p, y).
  • verdict: "beats_baseline" when accuracy is at least 10 points above majority, else "no_signal".

Round the numbers to 3 places.

3. Run. Without today, every coupon gets roughly 0.3 to 0.5, and accuracy equals the majority baseline. The AUC is around 0.75: Jev leans slightly toward "expired" for older dates, but no probability crosses 0.5, so as a yes/no answer it is useless. Add "today" to the state and it is right on every coupon, with probabilities near 0 and 1.

Starter file: coupons.jsonl

{"id": 1, "code": "SAVE10", "expires": "2025-08-20", "expired": true}
{"id": 2, "code": "SAVE11", "expires": "2026-03-08", "expired": true}
{"id": 3, "code": "SAVE12", "expires": "2026-05-27", "expired": true}
{"id": 4, "code": "SAVE13", "expires": "2026-07-26", "expired": true}
{"id": 5, "code": "SAVE14", "expires": "2026-08-25", "expired": true}
{"id": 6, "code": "SAVE15", "expires": "2026-09-10", "expired": true}
{"id": 7, "code": "SAVE16", "expires": "2026-09-17", "expired": true}
{"id": 8, "code": "SAVE17", "expires": "2026-09-21", "expired": true}
{"id": 9, "code": "SAVE18", "expires": "2026-09-23", "expired": true}
{"id": 10, "code": "SAVE19", "expires": "2026-09-25", "expired": false}
{"id": 11, "code": "SAVE20", "expires": "2026-09-26", "expired": false}
{"id": 12, "code": "SAVE21", "expires": "2026-09-29", "expired": false}
{"id": 13, "code": "SAVE22", "expires": "2026-10-03", "expired": false}
{"id": 14, "code": "SAVE23", "expires": "2026-10-14", "expired": false}
{"id": 15, "code": "SAVE24", "expires": "2026-11-08", "expired": false}
{"id": 16, "code": "SAVE25", "expires": "2026-12-23", "expired": false}
{"id": 17, "code": "SAVE26", "expires": "2027-02-21", "expired": false}
{"id": 18, "code": "SAVE27", "expires": "2027-07-21", "expired": false}
{"id": 19, "code": "SAVE28", "expires": "2026-01-17", "expired": true}
{"id": 20, "code": "SAVE29", "expires": "2026-06-26", "expired": true}
{"id": 21, "code": "SAVE30", "expires": "2026-08-10", "expired": true}
{"id": 22, "code": "SAVE31", "expires": "2026-09-14", "expired": true}
{"id": 23, "code": "SAVE32", "expires": "2026-09-27", "expired": false}
{"id": 24, "code": "SAVE33", "expires": "2026-10-06", "expired": false}
{"id": 25, "code": "SAVE34", "expires": "2026-10-24", "expired": false}
{"id": 26, "code": "SAVE35", "expires": "2026-11-23", "expired": false}
{"id": 27, "code": "SAVE36", "expires": "2027-04-12", "expired": false}
{"id": 28, "code": "SAVE37", "expires": "2027-09-24", "expired": false}
{"id": 29, "code": "SAVE38", "expires": "2026-09-19", "expired": true}
{"id": 30, "code": "SAVE39", "expires": "2026-10-01", "expired": false}
{"id": 31, "code": "SAVE40", "expires": "2026-09-22", "expired": true}
{"id": 32, "code": "SAVE41", "expires": "2026-09-28", "expired": false}
{"id": 33, "code": "SAVE42", "expires": "2026-03-28", "expired": true}
{"id": 34, "code": "SAVE43", "expires": "2027-01-22", "expired": false}
{"id": 35, "code": "SAVE44", "expires": "2026-09-03", "expired": true}
{"id": 36, "code": "SAVE45", "expires": "2026-10-09", "expired": false}
{"id": 37, "code": "SAVE46", "expires": "2025-10-29", "expired": true}
{"id": 38, "code": "SAVE47", "expires": "2027-05-22", "expired": false}
{"id": 39, "code": "SAVE48", "expires": "2026-09-16", "expired": true}
{"id": 40, "code": "SAVE49", "expires": "2026-10-04", "expired": false}

Prerequisites

  • Python with lists, dicts and numpy arrays
  • The Jev basics: a state, typed questions, noul and choice answers
  • No API keys: the lab's proxy provides access to Jev

Exam domains covered

EvaluationReliabilityDecision Models

Skills & technologies you'll practice

This intermediate-level ai/ml lab gives you real-world reps across:

JevTypeSafefailure modesmodel evaluationdecision modelsprompt injectionreliabilityintermediate

How to test a decision model before you rely on it

A decision model returns a probability for any question you ask, including questions it has no way to answer. The probability looks the same either way, so the failures do not show until users hit them. The fix is a set of probes you run before a question ships. In this lab you write those probes for Jev, TypeSafe's decision model, on four real feature ideas. You compare accuracy with a majority-class baseline to catch facts missing from the state. You measure confident wrong answers on arithmetic that code checks exactly and for free. You count off-topic messages that a choice question forces into a real option and fix them with an escape option. You test stability under reordered options and under text in the state that argues for the opposite answer. The probes feed one verdict per feature: ship, add the missing fact, compute it in code, add an "other" option, or treat it as unstable.

Frequently asked questions

Do I need a Jev or OpenRouter API key for this lab?

No. The lab sandbox reaches Jev through Preporato's model proxy, which holds the key.

Is Jev robust to prompt injection?

The lab measures one case: a review filter where a note inside the review argues for the opposite answer. In that probe the notes moved probabilities a little and flipped no decision. That is one measurement on one question, so the lab keeps the probe in the suite rather than treating it as a guarantee.

Why can't Jev check an invoice total?

Jev answers in a single fast pass without working through the numbers, and on invoices that do not add up it still reports high confidence. Checking arithmetic in code is exact and costs nothing, so that part of the decision belongs in code.

What is a closed-world choice question?

A choice question whose options do not cover every input. Jev has to pick one of the options you give it, so a message that fits none of them still lands in one, sometimes with high confidence. An explicit 'other' option gives those messages somewhere to go.