Debug With an Agent: Reproduce, Fix, and Add the Regression Test
Hands-on lab · IDE in your browser

Debug With an Agent: Reproduce, Fix, and Add the Regression Test

Build an agent that debugs a failing function the disciplined way: reproduce the failure before touching the code, prompt a model for a fix from the buggy source and the failing tests, verify the fix against the whole suite, loop while feeding failures back, and close the bug for good with a regression test that fails on the old code and passes on the new.

Time
55 min
Checked steps
5
Level
Intermediate
Setup
None
Read step 1

Hands-on labs require Pro · $29.99/mo · cancel anytime

The job

A bug report says median() returns the wrong answer for even-length lists. You build an agent that does what a careful engineer does: it reproduces the failure first, asks a model for a fix, verifies it against the whole test suite rather than just the failing case, retries with the failures fed back, and finishes by writing a regression test that would have caught the bug in the first place.

5 steps, each checked when you finish it

A check runs your work at the end of every step. Hints and the full solution are there if you get stuck.

  1. 1

    Reproduce the bug

    A bug report lands: median() returns the wrong answer for lists with an even number of items.

  2. 2

    Propose a fix

    Now bring in the model.

  3. 3

    Verify the fix

    A fix is a hypothesis until the tests say otherwise.

  4. 4

    The debug loop

    Put it together into the loop a developer runs by hand: reproduce, propose a fix, verify, and if it is still red, hand the failures back to the model and try again, up to a budget.

  5. 5

    A regression test

    The bug is fixed, but it is not closed.

Step 1 as it appears in the lab

The lab’s own text. The hint and the solution stay inside the lab.

Step 1: Reproduce the bug

A bug report lands: median() returns the wrong answer for lists with an even number of items. The code and its test suite are in harness.BUGGY and harness.TESTS. The first rule of debugging is the one everyone skips under pressure: reproduce the failure before you touch anything. A bug you cannot reproduce is a bug you cannot fix, and cannot prove you fixed.

Write two functions in agent.py:

  • extract_code(reply): pull code out of a model reply, taking the first ``` fenced block (dropping an optional python tag) or the whole reply when there is no fence. You will need it from Step 2 on.
  • reproduce(buggy_src, tests_src): run the suite with harness.run and return the failure record. If nothing fails, raise ValueError: there is no bug to reproduce, so there is nothing to debug.

Run it: two even-length tests fail, exactly the reported symptom.

agent.py, the file you edit48 lines
"""Your debugging agent. It reproduces a failure, asks a model for a fix, verifies the fix against the whole
suite, loops until green, and then insists on a regression test that fails on the old code, the discipline
that stops the same bug coming back."""
import harness


def extract_code(reply):
    """Pull code out of a model reply: the first ``` fenced block (dropping an optional python tag), or the
    whole reply stripped when there is no fence."""
    # TODO (Step 1): first ``` fenced block (drop a leading 'python'), else the whole reply stripped.
    raise NotImplementedError("Step 1: write extract_code()")


def reproduce(buggy_src, tests_src):
    """Run the suite against the buggy code and return the failure. Raise if nothing fails: you cannot fix a
    bug you cannot reproduce."""
    # TODO (Step 1): run the suite on the buggy code; return the failure, raise ValueError if nothing fails.
    raise NotImplementedError("Step 1: write reproduce()")


def propose_fix(buggy_src, failure, ask=harness.ask, feedback=""):
    """Ask the model to fix the code given the failing tests. Fold in feedback from a previous attempt.
    Return the extracted code."""
    raise NotImplementedError("propose_fix() arrives in Step 2")


def verify(fixed_src, tests_src):
    """Run the whole suite against a candidate fix. Returns {"passed", "failed", "total", "error", "ok"}
    where ok is True only when every test passes."""
    raise NotImplementedError("verify() arrives in Step 3")


def debug(buggy_src, tests_src, ask=harness.ask, max_iters=3):
    """Reproduce, then propose a fix and verify it, feeding the still-failing tests back until the suite is
    green or the budget runs out. Returns {"code", "fixed", "iters", "history"}."""
    raise NotImplementedError("debug() arrives in Step 4")


def guards_bug(test_src, buggy_src, fixed_src):
    """Does this test actually guard the bug? A real regression test fails on the buggy code and passes on
    the fixed code. Returns True only when both hold."""
    raise NotImplementedError("guards_bug() arrives in Step 5")


def add_regression_test(buggy_src, fixed_src, ask=harness.ask):
    """Ask the model for a new test_* that captures the bug, then keep it only if it actually guards the bug
    (fails on the old code, passes on the new). Returns {"test", "guards"}."""
    raise NotImplementedError("add_regression_test() arrives in Step 5")
Provided for you:harness.pytry_it.py

Frequently asked questions

What are the steps to debug code systematically?

Reproduce the failure reliably, localize the cause, apply the smallest fix that addresses it, verify against the full test suite rather than only the failing case, and add a regression test that fails on the old code so the bug cannot silently return.

Why write a regression test after fixing a bug?

A fix without a test that fails on the old code proves nothing and lets the same bug come back unnoticed. A regression test must fail on the buggy version and pass on the fixed version; if it passes on both, it does not guard the bug.

Can an AI agent debug code on its own?

An agent can propose fixes, but it needs a loop with verification: reproduce the failure, run the model's fix against the whole suite, feed any remaining failures back, and refuse to call the bug fixed while tests are red. The same discipline a human debugger uses is what keeps the agent honest.

Build an AI debugging agent that reproduces, fixes and guards

Debugging has a shape that does not change when a model does the typing: reproduce the failure, form a hypothesis, apply the smallest fix, verify against the whole suite, and add a test so the bug cannot return. Skip the first or last step and you get fixes that do not hold. In this lab you build a debugging agent around that discipline. It runs the failing suite to reproduce the bug, prompts a model for a fix from the code and the failures, verifies every test, loops with the failures fed back, and then keeps only a regression test that fails on the old code and passes on the new.