Preporato
Prompting That Passes Tests: Improve a Prompt with a Score, Not a Hunch
Hands-on lab · IDE in your browser

Prompting That Passes Tests: Improve a Prompt with a Score, Not a Hunch

Improve a real support-triage prompt against a 20-case test set: state an exact output format, write the policy as rules, fence off customer text against prompt injection, add few-shot examples aimed at the cases that still fail, and confirm the result on a held-out set you never tuned on.

Time
45 min
Checked steps
5
Level
Beginner
Setup
None
Read step 1

Hands-on labs require Pro · $29.99/mo · cancel anytime

Lab cockpit45 min · 5 stepsSession running
2 / 5 steps passingFence off the data · step 3 of 5
prompt.md▶ Run✓ Check
- complaint: unhappy about staff, the shop, or a wrong or damaged item, without asking for money back- other: anything else, including thanks, praise and stock questions How to choose the priority:- high: the customer was charged wrongly, an item is damaged, wrong or lost, or they need something within two days- low: the message asks for nothing (thanks, praise or feedback only)- normal: everything else Order id: BL- followed by exactly 5 digits, copied exactly. Phone numbers, ISBNs and other numbers are not order ids. If there is no valid order id, write none. ---USER--- {message}
TerminalOutput

The job

Brightline Books gets about eighty customer messages a day, and support wants each one tagged with a category, a priority and the order id before anyone reads it. You write the prompt that does it. A harness runs your prompt over twenty labelled messages on a live model and scores every field, so each change you make shows up as a number. Some messages hide deadlines, some carry phone numbers and ISBNs that look like order ids, and three try to give the model orders of their own.

5 steps, each checked when you finish it

A check runs your work at the end of every step. Hints and the full solution are there if you get stuck.

  1. 1

    Say exactly what you want back

    Brightline gets about eighty customer messages a day.

  2. 2

    Write the rules down

    The model cannot read your mind or Brightline's support policy.

  3. 3

    Fence off the data

    Look at cases #12, #13 and #19.

  4. 4

    Show examples of the hard cases

    Rules cover the general case; examples settle the borderline ones.

  5. 5

    The held-out test

    You have tuned the prompt by looking at 20 cases again and again, so some of its improvement is fitted to *those* messages.

Step 1 as it appears in the lab

The lab’s own text. The hint and the solution stay inside the lab.

Step 1: Say exactly what you want back

Brightline gets about eighty customer messages a day. They want a model to read each one and fill in three fields for the support queue: a category, a priority and the order id, if there is one.

This lab has no Python to write. You edit prompt.md, and a harness runs it over 20 real-looking messages in cases.jsonl, each with the right answer, then scores the replies. That is how prompts are improved at work: against a test set, with numbers, rather than by eye on one example.

prompt.md has two parts, split by the line ---USER---:

  • above it, the system prompt: the standing instructions;
  • below it, the user template: {message} is replaced by each customer message.

Text between <!-- and --> is a note for you; the harness removes it.

Do this

1. Click Run first, with the prompt as it is. Every case fails: the model writes a friendly paragraph, and a program cannot read a paragraph.

2. Replace the TODO with the output contract. Say it plainly:

Reply with exactly one line in this format and nothing else:
category=<category>; priority=<priority>; order=<order id or none>

category is one of: order_status, refund, recommendation, event, complaint, other
priority is one of: high, normal, low

3. Run again. Nearly every reply now parses, and some are already right. Read the failures: the model is guessing what the categories mean and what makes a message urgent. Step 2 fixes that.

prompt.md, the file you edit8 lines
<!-- Text inside these arrows is a note for you. The harness removes notes before sending. -->
You triage customer messages for Brightline Books, an independent bookshop.
<!-- TODO (Step 1): tell the model exactly what to reply:
     one line, the format  category=<category>; priority=<priority>; order=<order id or none>
     and the allowed values for category and priority. See the instructions. -->
Classify this customer message.
---USER---
{message}
Provided for you:cases.jsonlholdout.jsonlpromptlab.pyrun_tests.py

Frequently asked questions

Do I need to code for this lab?

No. You edit prompt.md, a plain text file with a system prompt and a user template, and click Run. A harness sends it to a hosted model for every test case and prints a scorecard.

What is few-shot prompting?

Including a few worked examples, each an input and the exact output you want, in the prompt. Examples settle borderline cases that rules leave open. The lab has you write examples aimed at the cases your rules still get wrong, in your own words rather than copied from the test set.

Do delimiters stop prompt injection?

They reduce it but do not stop a determined attacker. Wrapping outside text in tags, telling the model it is data, and repeating the task after it defeats the simple injections in this lab on a small model; the output is also parsed strictly so an injected answer cannot add fields.

Why is there a held-out set?

Tuning a prompt on the same cases you score it on fits it to those cases. A separate set that you never tune on gives the honest number. Moving a held-out case into your examples ends its value as a test.

Why prompts should be improved against a test set

A prompt that looks right on the example you tried can fail on a third of real inputs, and a change that fixes one case often breaks another. Prompt engineering in production means writing labelled test cases first and changing the prompt only when the score says it helped. In this lab you improve one classification prompt through the standard techniques, and measure each: an explicit output contract that a parser can read, written rules for every label, delimiters and a closing instruction that blunt prompt injection, few-shot examples aimed at the cases that still fail, and a held-out set that shows how much of your gain was fitted to the cases you tuned on.