Engineer an Extraction Prompt and Prove It on a Holdout Set with Headless Claude
Write one prompt that turns messy customer-support emails into strict JSON (intent, urgency, order id, deadline), then prove it on 8 holdout emails it has never seen. You run the cases through your own Claude Code in headless mode (claude -p), score them against hashed ground truth (32 field checks, pass at 27), and iterate on explicit criteria, few-shot examples, and XML structure until the eval goes green. The holdout emails carry real traps: stale order ids in forwarded threads, an urgency flip mid-email, refund wording that is actually a complaint, and relative dates you resolve from the Date: header.
1 hr
Est. time
4
Outcomes
5
Rubric criteria
65%
Pass score
What you'll learn
Skills you'll have real reps in after shipping this.
The scenario
Your team wants to auto-triage the support inbox: every incoming email should become a JSON record with the customer's intent, how urgent it is, the order id it concerns, and any reply-by deadline. A first attempt ('extract the fields from this email') works on the easy messages and falls apart on the real ones: customers quote old order ids in forwarded threads, write 'no rush' two lines before announcing a hard deadline, and say the word refund while asking for something else entirely.
This task is that job, done properly. You get 4 dev emails with their expected JSON (your few-shot material) and 8 holdout emails with no answers. You engineer the prompt: explicit criteria for every field, worked examples that teach the hard calls, XML structure so instructions and input stay separate, and a strict output format. Then you prove it: a run script pipes each holdout email through your own Claude Code in headless mode, and a local scorer checks all 32 field values against hashed ground truth. The evidence file is written only when the eval genuinely passes.
Your role
You are the engineer who owns the extraction prompt for a support-triage pipeline. Your deliverable is the prompt itself, the real model outputs it produced on the holdout set, machine-generated eval evidence that at least 27 of 32 field checks hit, and a short note on what failed first and how you fixed it, packaged as a single submission.
Start the task to unlock the full brief
You'll get the step-by-step requirements, setup commands, the 5-criterion grading rubric, tips, and the ability to submit your solution for instant AI grading.
Free to start · submit when you're ready
Prove your prompt works before you trust it, hands-on
Anyone can write a prompt that handles the easy cases; this task makes you engineer one that survives a holdout set. You take a messy support-inbox extraction job (intent, urgency, order id, deadline), study 4 dev emails with known answers, and then earn 27 of 32 field checks on 8 emails your prompt has never seen, with the cases running through your own Claude Code in headless mode (claude -p). Along the way you practice the mechanics that make prompts reproducible: explicit per-field criteria, few-shot examples chosen to cover the hard calls, XML structure that separates instructions from input, and a strict output format, plus a scripted eval loop you can reuse on any extraction problem.