AI Code Review Bot: Parse a Diff, Find Bugs, Score and Gate the PR
Hands-on lab · IDE in your browser

AI Code Review Bot: Parse a Diff, Find Bugs, Score and Gate the PR

Build a code review bot around a model. Parse a pull-request diff into the lines that changed, prompt the model for structured findings and read them back through prose and code fences, score what it caught against the bugs planted in the fixtures, triage the noise into a ranked list, and turn it into a block-or-approve verdict with a review comment.

Time
50 min
Checked steps
5
Level
Intermediate
Setup
None
Read step 1

Hands-on labs require Pro · $29.99/mo · cancel anytime

The job

Your team wants a bot that reviews every pull request and flags the dangerous changes before a human looks. You build it: it reads the diff, asks a model to find real bugs in the added lines, grades itself against known-planted bugs so you can trust it, cuts the noise, and leaves a verdict and a comment on the PR.

5 steps, each checked when you finish it

A check runs your work at the end of every step. Hints and the full solution are there if you get stuck.

  1. 1

    Parse the diff

    A review bot works on a pull request's diff.

  2. 2

    Ask the model to review

    Now hand the changed lines to the model and get back structured findings.

  3. 3

    Score against the planted bugs

    To know whether the bot is any good, grade its findings against the bugs the fixtures planted.

  4. 4

    Triage the findings

    A raw model review is noisy: low-value nits and the same issue reported twice.

  5. 5

    Verdict and comment

    The bot ends where a human reviewer does: a decision and a comment.

Step 1 as it appears in the lab

The lab’s own text. The hint and the solution stay inside the lab.

Step 1: Parse the diff

Parse the diff

A review bot works on a pull request's diff. Before you can ask a model about it or grade what it finds, you have to turn the raw unified diff into the lines that actually changed, each with its line number in the new file, the number everyone will refer to.

harness.py gives you ask(prompt) (the model, cached) and load_diffs() (the fixtures: each has a diff and the bugs planted in it). You write your bot in reviewer.py.

Write parse_diff(diff_text): return [{"file", "added": [{"line", "code"}]}], one entry per file, where line is the added line's number in the new file. Read the new-file start from each @@ -a,b +c,d @@ header; an added (+) line sits at the current number and advances it, a context ( ) line advances it, a removed (-) line does not.

reviewer.py, the file you edit77 lines
"""Your code review bot. It reads a pull-request diff, asks the model to find bugs in the added lines,
scores what it found against the bugs planted in the fixtures, drops the noise, and turns the result into a
merge decision and a review comment."""
import json

import harness

SEVERITY = {"low": 1, "medium": 2, "high": 3}


# ---------- Step 1: parse the diff ----------

def parse_diff(diff_text):
    """The added lines of a unified diff, per file. Return [{"file", "added": [{"line", "code"}]}] where
    line is the added line's number in the NEW file. Track the new-file counter from each @@ hunk header;
    added ('+') and context (' ') lines advance it, removed ('-') lines do not."""
    # TODO (Step 1): walk the diff; set the new-file counter from each @@ header; a '+'
    # line is added at the counter (then +1), a ' ' context line advances it, a '-' line does not.
    raise NotImplementedError("Step 1: write parse_diff()")


# ---------- Step 2: ask the model to review ----------

def review(diff_text, ask=harness.ask):
    """Ask the model to review the diff and return its findings as
    [{"file", "line", "severity", "issue"}]. Present the added lines with their numbers, require a JSON
    array back, and parse it even when the model wraps it in prose or a code fence. On unparseable output
    return []."""
    raise NotImplementedError("review() arrives in Step 2")


def _parse_findings(text):
    """Pull the JSON array of findings out of the model's reply; [] if there is none."""
    start, end = text.find("["), text.rfind("]")
    if start < 0 or end < start:
        return []
    try:
        data = json.loads(text[start:end + 1])
    except json.JSONDecodeError:
        return []
    out = []
    for d in data:
        if isinstance(d, dict) and "file" in d and "line" in d:
            out.append({"file": d["file"], "line": int(d["line"]),
                        "severity": d.get("severity", "medium"), "issue": d.get("issue", "")})
    return out


# ---------- Step 3: score against the planted bugs ----------

def score(findings, bugs, window=2):
    """Precision, recall and F1 of findings against the planted bugs. A finding matches a bug when they are
    in the same file and within `window` lines; each bug counts at most once. Returns
    {"tp", "fp", "fn", "precision", "recall", "f1"}."""
    raise NotImplementedError("score() arrives in Step 3")


# ---------- Step 4: triage the findings ----------

def triage(findings, min_severity="medium"):
    """Cut the noise: drop findings below min_severity, and when several land on the same file and line keep
    only the most severe. Return the survivors, most severe first."""
    raise NotImplementedError("triage() arrives in Step 4")


# ---------- Step 5: the verdict ----------

def verdict(findings):
    """The merge decision: block when any high-severity finding remains. Return
    {"block": bool, "high": <count>}."""
    raise NotImplementedError("verdict() arrives in Step 5")


def summary(findings):
    """A short review comment: one line per finding, most severe first, or an approval line if there are
    none."""
    raise NotImplementedError("summary() arrives in Step 5")
Provided for you:diffs.jsonharness.pytry_it.py

Frequently asked questions

How do you get structured output from a code review model?

Ask for a JSON array with fixed keys and parse it defensively: models often wrap JSON in prose or a code fence, so locate the array within the reply and fall back to an empty result on malformed output rather than trusting the text verbatim.

How do you measure whether a review bot is any good?

Grade its findings against known bugs. Match each finding to a planted bug in the same file within a small line window, count each bug once, and compute precision and recall. Recall is how many real bugs it caught; precision is how much noise came with them.

Why triage a model's code review findings?

Raw model reviews include low-value nits and the same issue reported more than once. Triage drops findings below a severity threshold and collapses duplicates on the same line, leaving a short ranked list a reviewer will actually act on.

Building an AI code review bot

A model can spot a SQL injection or an off-by-one in a diff, but a useful review bot is mostly the plumbing around it: parsing the diff, getting structured findings out of a chatty reply, measuring whether the findings are any good, and cutting the noise into a decision. In this lab you build that bot end to end. You parse a unified diff, prompt a model for JSON findings and parse them robustly, score the findings against planted bugs with precision and recall, triage duplicates and low-severity noise, and produce a block-or-approve verdict with a review comment.