Preporato
Spec to Code: Build a Test-Driven Coding Agent
Hands-on lab · IDE in your browser

Spec to Code: Build a Test-Driven Coding Agent

Build the loop a coding agent runs: turn a specification into tests, generate a solution, run the tests in a sandbox, feed the failures back and try again until they pass, then check the result against held-out tests so a green run means the problem was solved and not gamed.

Time
55 min
Checked steps
5
Level
Intermediate
Setup
None
Read step 1

Hands-on labs require Pro · $29.99/mo · cancel anytime

The job

You are building the engine behind a coding assistant: give it a specification and it should produce working code on its own. It writes the tests, writes a solution, runs the tests, and repairs its own failures until they pass, then proves the solution holds up on cases it never saw.

5 steps, each checked when you finish it

A check runs your work at the end of every step. Hints and the full solution are there if you get stuck.

  1. 1

    Read the model's code

    A coding agent runs a loop a developer knows well: write tests, write code, run the tests, read the failures, try again.

  2. 2

    Write tests from the spec

    Tests come first: they are how the agent will know when it is done.

  3. 3

    Write a solution

    Now ask the model to implement the function, and make the prompt able to carry feedback, because the whole point of the loop is that a failed attempt teaches the next one.

  4. 4

    The agent loop

    This is the agent.

  5. 5

    Guard against overfitting

    A green test run is only trustworthy if the tests were fair.

Step 1 as it appears in the lab

The lab’s own text. The hint and the solution stay inside the lab.

Step 1: Read the model's code

Read the model's code

A coding agent runs a loop a developer knows well: write tests, write code, run the tests, read the failures, try again. You will build that loop. It starts with the dullest but most necessary piece, getting runnable code out of a model reply, because everything downstream runs what the model returns.

harness.py gives you ask(prompt) (the model, cached), SPEC (the task: a leap-year function), HOLDOUT_TESTS, and run(solution_src, test_src) which executes tests against a solution in a separate process and reports {"passed", "failed", "total", "error"}. You write the agent in agent.py.

Write extract_code(reply): if the reply contains a ``` fence, return the first fenced block (dropping a leading python tag); otherwise return the whole reply, stripped. Run it on a fenced and a bare reply.

agent.py, the file you edit46 lines
"""Your coding agent. Given a spec, it writes tests, writes a solution, runs the tests, and feeds the
failures back to itself until the tests pass, the loop a developer runs by hand. Then it checks the result
against held-out tests to be sure the solution solves the problem beyond the visible examples."""
import harness


# ---------- Step 1: read the model's code ----------

def extract_code(reply):
    """Pull the code out of a model reply. If it is fenced in ``` (optionally ```python), return the first
    fenced block; otherwise return the whole reply, stripped."""
    # TODO (Step 1): if the reply has a ``` fence, return the first fenced block (drop a
    # leading 'python' tag); otherwise the stripped reply.
    raise NotImplementedError("Step 1: write extract_code()")


# ---------- Step 2: write tests from the spec ----------

def write_tests(spec, ask=harness.ask):
    """Ask the model for test functions (named test_*) that encode the spec, and return the code. The tests
    call the function directly, so they do not import it."""
    raise NotImplementedError("write_tests() arrives in Step 2")


# ---------- Step 3: write a solution ----------

def write_solution(spec, ask=harness.ask, feedback=""):
    """Ask the model to implement the function. If feedback from a failed run is given, include it so the
    model can fix its previous attempt. Return the code."""
    raise NotImplementedError("write_solution() arrives in Step 3")


# ---------- Step 4: the agent loop ----------

def solve(spec, ask=harness.ask, max_iters=4):
    """Write tests once, then write a solution and run it, feeding the failures back and trying again until
    the tests pass or max_iters is reached. Return {"code", "tests", "passed", "iters", "history"}."""
    raise NotImplementedError("solve() arrives in Step 4")


# ---------- Step 5: guard against overfitting ----------

def holdout_check(solution_src):
    """Run the held-out tests the agent never saw against a solution, and return the names of any that
    fail. An empty list means the solution generalises beyond the visible examples."""
    raise NotImplementedError("holdout_check() arrives in Step 5")
Provided for you:harness.pytry_it.py

Frequently asked questions

How does a coding agent use test failures?

It feeds them back into the next prompt. After running the generated tests, the agent includes the names or output of the failing tests in the prompt for the next attempt, so the model can repair its previous solution rather than starting blind. The loop repeats until the tests pass or a limit is reached.

Why run generated code in a separate process?

Isolation and safety. Running an unknown solution and its tests in a fresh subprocess with a timeout keeps a crash, an infinite loop or a stray side effect from taking down the agent, and gives a clean pass or fail result to drive the loop.

How do you stop a coding agent from gaming the tests?

Grade the final solution on held-out tests it never saw. A solution that hardcodes the visible examples passes the tests it was shown but fails the held-out ones, which turns a falsely green run red.

Building a test-driven coding agent

A coding agent is a loop: write tests from the spec, write code, run the tests, read the failures, and try again. The model does the writing, but the loop, the sandbox that runs the code, and the check that the solution generalises are what make it reliable. In this lab you build that agent. You extract runnable code from a model reply, generate tests from a spec, generate a solution that can take feedback, drive the write-run-repair loop to a passing state, and guard against a solution that only fits the visible examples with held-out tests.