Preporato
Compare AI Models Before You Trust One: Accuracy, Cost and a Cascade
Hands-on lab · IDE in your browser

Compare AI Models Before You Trust One: Accuracy, Cost and a Cascade

Choose between a small and a large language model on numbers instead of reputation, without code: measure accuracy, cost and speed on labelled return requests, repeat the runs to see how much results wobble, turn a manager's memo into a measurable bar, find where the cheap model is reliable, and build a cascade that meets both the accuracy and the budget, confirmed on next month's requests.

Time
40 min
Checked steps
4
Level
Beginner
Setup
None
Read step 1

Hands-on labs require Pro · $29.99/mo · cancel anytime

Lab cockpit40 min · 4 stepsSession running
2 / 4 steps passingA cascade: small first, large when it matters · step 3 of 4
compare.yaml▶ Run✓ Check
#   models:  which models to run on every request: small (Llama 3.1 8B) and/or large (Llama 3.3 70B)#   runs:    how many times to repeat each measurement (1 or 2)#   bar:     the requirements from memo.md, as numbers#   cascade: small model first, and the decisions it makes that go to the large model insteadmodels: [small, large]runs: 2bar:  min_accuracy: 0.85  max_usd_per_1000: 0.08
TerminalOutput

The job

Brightline's returns desk wants a model to make the first decision on each return request. A small model is cheap and a large one is about ten times the price. The operations manager wants 85 in 100 decisions right and a model bill under 16 cents a month. You measure both, discover that neither meets the bar alone, and build a cascade that does.

4 steps, each checked when you finish it

A check runs your work at the end of every step. Hints and the full solution are there if you get stuck.

  1. 1

    Measure both models

    Brightline's returns desk wants a model to make the first decision on each return request, using the policy in policy.md.

  2. 2

    Write the bar down

    "Better" is not a decision.

  3. 3

    A cascade: small first, large when it matters

    The small model misses the accuracy bar and the large one misses the budget.

  4. 4

    Next month's requests

    A cascade tuned on 40 requests has to hold on the next 40.

Step 1 as it appears in the lab

The lab’s own text. The hint and the solution stay inside the lab.

Step 1: Measure both models

Brightline's returns desk wants a model to make the first decision on each return request, using the policy in policy.md. The prompt is written (prompt.md). The open question is which model to use: a small one (Llama 3.1 8B) or a large one (Llama 3.3 70B), which costs about ten times as much per token.

Pick on numbers, not reputation. requests.jsonl holds 40 real-looking requests, each with the decision the policy gives. You measure each model's accuracy, cost and speed on them. There is no code: you edit compare.yaml.

One run is not enough. The same model on the same requests at temperature 0 can still give different answers from one run to the next, because the hosted service routes calls to different servers. A difference smaller than that wobble means nothing.

Do this:

  1. Answer the question below first.
  2. In compare.yaml, set models: [small, large] and runs: 2.
  3. Click Run (about a minute). Read the table: accuracy, cost per 1,000 requests, and speed (p50 is the typical time, p95 the slowest 1 in 20). Compare the spread between runs with the gap between the models.

The check reads the measurements your Run saved.

compare.yaml, the file you edit7 lines
# compare.yaml: what to measure. Run passes this to the harness.
#   models:  which models to run on every request: small (Llama 3.1 8B) and/or large (Llama 3.3 70B)
#   runs:    how many times to repeat each measurement (1 or 2)
#   bar:     the requirements from memo.md, as numbers
#   cascade: small model first, and the decisions it makes that go to the large model instead
models: [small]    # TODO (Step 1): measure both models, twice
runs: 1
Provided for you:compare.pymemo.mdnew_requests.jsonlpolicy.mdprompt.mdrequests.jsonlrun_compare.py

Frequently asked questions

Do I need to code for this lab?

No. You edit compare.yaml, a short settings file, and click Run. A harness runs the models and prints accuracy, cost and speed.

Why run the same test twice?

Hosted models can give different answers on identical runs. In the lab the small model moves by several points between runs; a gap smaller than that between two models means nothing.

What is a model cascade?

A cheap model handles every request first, and only the answers it is unreliable on are sent to a stronger model. In the lab this meets an accuracy bar the small model misses at a cost the large model cannot match.

How do you decide which answers to escalate?

Measure how often the small model is right for each kind of answer it gives. Keep the answers it is reliable on and escalate the rest, plus any it could not decide.

Choosing between language models with measurements

Which model should an automation use? The honest answer comes from measuring candidates on your own task: accuracy against labelled examples, cost per thousand requests, speed, and how much results move between identical runs. In this lab you compare a small and a large open model on return requests without writing code. You repeat each measurement to see the noise, write the business requirements as a bar, read where the cheap model is reliable and where it is not, and route only the unreliable decisions to the large model, confirmed on new requests.