TL;DR: In Preporato's AI study tutor, one component picks the tutor's next move from a closed set of six. We moved that decision from an LLM agent loop to Jev, TypeSafe's decision model, and measured both on real states: Jev matched the written rules 20 times out of 20 against 13 for the loop, gave the same move every time a state repeated, and answered in a median 520 ms against 18 seconds. Jev failed on three other jobs we tried: predicting what a learner would do next (worse than guessing the most common action), spotting hints that give an answer away, and one intent it routed wrongly at 0.94 confidence. The pattern behind both lists is the rule we now use: Jev answers narrow questions about what is present in its input, code enforces preconditions, and anything that needs imagination or text goes to an LLM.
This comparison comes from a system with users. Preporato's study tutor runs a session with a learner: it chooses questions, reacts to answers, and decides when to step in. The part that decides when to step in is called the director, and until September it was the most expensive part of a session. That made it the natural place to try Jev, and it gave us a before-and-after on the same real inputs.
Try the two sides yourself
The Jev learning hub has a live demo and a roadmap from first call to production. The free Jev tool-call gate lab reproduces the pattern from this article on an AI agent: Jev's answers, thresholds in code, and rules for the facts Jev cannot see.
The job we gave Jev
After each answer a learner gives, the director looks at the session state (the misses so far, the current streak, which topics failed, what has already been offered) and picks one of six moves: stay out of the way, offer to cut the session short, reteach a topic, ask the learner to explain a concept back, propose a set of questions on their weak spots, or suggest a reread. The language model then writes the one sentence the tutor says.
Before Jev, the director was a bounded tool-calling loop on GLM 5.3 Flash. It could look up the topic outline, propose a move, and correct itself when a move was refused. It worked, and it was the most expensive part of the session: across five scripted learner journeys it accounted for 44.6 of the 62 seconds spent waiting on models, with a median of 4.5 seconds per call.
Picking one move out of six from numeric state is a classification problem. Jev is built for exactly that: you send the state and a choice question with the six moves as options, and it returns a pick with probabilities. The language model still writes the sentence, because Jev cannot produce text.
How we measured
Hand-written test cases flatter every decider, so we captured ten real director states from the lab's recorded journeys and ran each one twice through both deciders, interleaved so that any provider slowdown hit both equally. Each decision was graded against the director's written rules, the conditions under which each move is allowed. A move that breaks its own rule counts as wrong even if a human might find it reasonable.
| LLM agent loop | Jev | |
|---|---|---|
| Matches the written rules | 13/20 | 20/20 |
| Same move when a state repeats | 7/10 | 10/10 |
| Median latency | 18,051 ms | 520 ms |
On the full seam, back to back on twelve real states, total time went from 84.9 seconds to 16.5, the median from 6.4 seconds to 582 ms, and the worst case from 16.7 seconds to 5.3. In a complete run of the scripted journeys, the director went from 10 calls taking 46.3 seconds to 13 calls taking 8.3, and the number of friction findings in the run dropped from 15 to 8.
Latency did not grow with the number of questions. On a 2,300-token state, one question took 547 ms and forty took 553 ms. Eight concurrent requests of forty questions each finished in 624 ms of wall time. Each extra judgment an LLM makes is more tokens to generate, while for Jev it is close to free, and that changes how you design: ask each thing you want to know as its own narrow question and combine the answers in code.
The bug the measurement found
The first run did not look like the table above. Both deciders scored 11 out of 20, and they failed in opposite directions.
The LLM loop offered a targeted question set to a learner on a six-miss streak, and again to one on a seven-miss streak, although the rule for that move excludes long streaks. Jev proposed a reread after a single miss two questions into a session, which its rule also excludes. Both failures were mechanical: a written precondition, checkable in one line of code, was ignored.
So we enforced the preconditions in code. A function checks each proposed move against its written condition, and both deciders run through it. The loop went from 11 to 13. Jev went from 11 to 20.
The difference in how much each gained is the most useful finding in this comparison. With the hard constraints enforced outside the model, Jev's remaining job was judging which allowed move fits the state, and it did that consistently. The loop's remaining errors all passed the precondition check, so they were judgment calls between allowed moves. If you evaluate Jev, evaluate it with your preconditions in code. Asking any model to both respect hard rules and make a soft judgment in one pass measures the wrong thing.
Where Jev failed
We tried Jev on three more jobs in the same product. It lost on all three, and the reasons are more useful than the scores.
Predicting the learner's next action. To prefetch content, we asked Jev to predict what the learner would do next from the session so far. It scored 32 out of 64 on real transitions. Always predicting the most common next action scores 59%, so Jev did worse than the simplest baseline. The answer to this question lives in the future and depends on a person's intentions, and nothing in the state determines it.
Guarding hints against giving the answer away. The tutor writes hints, and a hint must not reveal the answer. We asked Jev whether each hint leaked. No threshold on its probability separated real giveaways from safe hints, including after a retry with explicit criteria for both. Deciding whether a hint leaks means imagining what a learner could infer from it, a counterfactual about a mind that is not in the state.
Routing a learner's message. A separate router classifies what the learner typed: an answer, a request to change the session, a question. Jev routed 14 of 15 real messages correctly against 15 of 15 for the shipping router. The one miss was "shorter please", which Jev classified as a question to answer, at 0.94 confidence. A router that is right 93% of the time can be fine, but a confidently wrong answer is hard to live with, because a threshold on confidence would not have caught it. We kept the LLM router, and will revisit it with a reliability curve over a few hundred labelled messages.
Calibration: how to use the confidence
Jev's probabilities are calibrated in aggregate, and that is the claim TypeSafe makes: across many answers given at 0.8, about 80% are right. The "shorter please" miss shows what calibration does not promise, which is that any single answer at 0.94 is right. Design for the confident miss.
The way to do that is a reliability check on your own data before you trust a threshold:
- Label a few hundred real inputs for the decision you care about.
- Run Jev on all of them and bucket the answers by confidence: 0.5 to 0.6, 0.6 to 0.7, and so on.
- For each bucket, compare the share of correct answers with the bucket's confidence.
- Pick an abstain band: the confidence range where accuracy is below what you can accept. Inside it, send the input to a fallback, an LLM or a person. Above it, act on Jev's answer.
Two details matter in practice. For a noul question, the probability itself is the confidence, so measure it as distance from 0.5. And for a choice, a low confidence often means the probability is split between two options, which is information: a support ticket split between billing and account can go to both queues.
Routing between a small and a large model
The same idea gives the cheapest win Jev offers in an LLM application. Before calling a model, ask Jev how hard the request is, as a score or a choice over difficulty tiers, and send easy requests to a small model and hard ones to a large model. A Jev call costs a fraction of a cent and a fraction of a second, so it pays for itself when it keeps a meaningful share of requests off the large model. Measure it the same way as any router: accuracy on a labelled set, per tier, with an abstain band that defaults to the large model.
When to use Jev and when to use an LLM
| The job | Use | Why |
|---|---|---|
| Pick one option from a fixed set, based on the input | Jev | Typed answer, stable across repeats, half a second |
| Several yes/no judgments about the same input | Jev | Latency is flat in the number of questions |
| Score against a rubric whose criteria are visible in the text | Jev | Calibrated probability per level |
| Enforce a hard precondition | Code | A rule never drifts and needs no threshold |
| Anything that needs text | LLM | Jev cannot generate text |
| Predict what someone will do | LLM, or neither | The answer is outside the state |
| Judge what someone could infer | LLM | A counterfactual, outside the state |
| A decision where one confident miss is expensive | Jev with an abstain band, or an LLM | Calibration is a claim about averages |
The rule we use
Ask Jev narrow questions about things that are present in the state, keep hard preconditions in code, and send low-confidence answers to a fallback. Treat every Jev failure, a timeout or an error, as no opinion and take the path you would take without it. With those three habits Jev replaced the slowest part of our tutor, and without them it scored the same 11 out of 20 as the LLM it replaced.
Frequently asked questions
Is Jev better than an LLM for classification?
For classifying a property that is present in the input, in our production test yes: 20 of 20 decisions matched the written rules against 13 of 20 for an LLM agent loop, at a median of 520 ms against 18 seconds. For questions whose answer depends on the future or on what someone could infer, it did worse than an LLM and in one case worse than always guessing the most common answer.
Can Jev replace an LLM-as-judge?
For rubric scoring on criteria that are visible in the text being judged, it is a strong candidate, and it is far faster and cheaper. For judgments that need reasoning about what a reader could infer, such as whether a hint gives an answer away, our tests found no usable threshold. Validate it on labelled examples from your own data before switching.
Are Jev's confidence scores reliable?
TypeSafe calibrates Jev so that answers given at a confidence are right about that often on average. Individual answers can still be confidently wrong: one message in our router test was misclassified at 0.94. Build an abstain band from a reliability check on your own labelled data.
How much does Jev cost compared with an LLM?
Jev charges $0.042 per million input tokens, and output is free. A decision with three questions about a short input costs around $0.00002. Adding questions barely changes the cost or the latency, which is the main difference from an LLM judge.
What should stay in code when using Jev?
Every precondition you can check mechanically: allowed moves for a given state, paths and hosts an agent may touch, required fields. In our tutor, moving those checks into code took Jev from 11 of 20 correct decisions to 20 of 20.
Sources: