Preporato
RAG Chunking Shoot-Out: Measure Which Chunking Strategy Retrieves
Hands-on lab · IDE in your browser

RAG Chunking Shoot-Out: Measure Which Chunking Strategy Retrieves

Measure chunking strategies for retrieval-augmented generation instead of guessing: write a recall metric over questions with exact evidence sentences, then compare whole documents, fixed-size chunks with and without overlap, heading-aware section chunks and sections with their heading path, on recall and on characters sent, and prove your pick on questions it has never seen.

Time
50 min
Checked steps
5
Level
Intermediate
Setup
None
Read step 1

Hands-on labs require Pro · $29.99/mo · cancel anytime

Lab cockpit50 min · 5 stepsSession running
1 / 5 steps passingFixed-size chunks · step 2 of 5
shootout.py▶ Run✓ Check
def whole_docs(docs):    """No chunking at all: each document is one chunk."""    return [{"doc": name, "path": name, "text": text} for name, text in docs.items()]  # ---------- Step 2: fixed-size chunks ----------def fixed_chunks(docs, size=400, overlap=0):    """Cut each document every `size` characters. With overlap, each chunk starts `size - overlap`    characters after the previous one, so neighbouring chunks share `overlap` characters."""
TerminalOutput

The job

Brightline Books is building a help assistant that answers from its own documents: the Book Box subscription terms, the shop guide and the events guide. Four subscription boxes each have their own Price, Skipping and Cancelling sections, which is exactly the kind of text naive chunking gets wrong. You build a measuring stick, run a shoot-out between chunking strategies and pick the one to ship on evidence.

5 steps, each checked when you finish it

A check runs your work at the end of every step. Hints and the full solution are there if you get stuck.

  1. 1

    A measuring stick

    Brightline Books wants a help assistant that answers from its own documents: the Book Box terms, the shop guide and the events guide in docs/.

    You writehit()evaluate()
  2. 2

    Fixed-size chunks

    The simplest chunking cuts every size characters.

    You writefixed_chunks()
  3. 3

    Follow the headings

    These documents already say where one idea ends: every section has a markdown heading.

    You writesection_chunks()
  4. 4

    Give every chunk its context

    Section chunks miss questions like "How much does the Kids Box cost?" The answer's chunk says "The Kids Box costs £12.99 a month", but four boxes each have a Price section.

    You writewith_context()
  5. 5

    Pick one and prove it

    You have measured five strategies on 30 questions.

    You writebest()

Step 1 as it appears in the lab

The lab’s own text. The hint and the solution stay inside the lab.

Step 1: A measuring stick

Brightline Books wants a help assistant that answers from its own documents: the Book Box terms, the shop guide and the events guide in docs/. Before the model answers, a retriever picks the pieces of text most similar to the question and sends only those. That is retrieval-augmented generation, or RAG. How you cut the documents into pieces, called chunks, decides what the retriever can find.

rag.py does the searching. It turns each chunk and each question into an embedding with a hosted model and returns the closest chunks. questions.json has 30 real questions, each with the evidence: the exact sentence that answers it. A chunk strategy is good if the evidence lands in the retrieved chunk, and if it sends the model little text besides.

You need a measuring stick before you compare anything.

Do this

1. Write hit(chunk_text, evidence): True when the whole evidence is inside the chunk, ignoring case and line breaks. normalise() is written for you.

2. Write evaluate(chunks, questions). For each question, ranked holds the indices of the retrieved chunks. Return:

  • recall: the share of questions with the evidence in a retrieved chunk;
  • chars_per_question: the average number of characters sent to the model;
  • chunks and missed.

3. Run. It measures the simplest strategy: no chunking, each whole document is one chunk.

The lab retrieves K = 1 chunk per question. With only about 40 chunks in these documents, one chunk is as strict as taking the top 3 out of the thousands a real knowledge base holds.

shootout.py, the file you edit64 lines
"""Chunking shoot-out: four ways to cut the same documents, measured on the same questions.
A chunk is a dict: {"doc": file name, "path": where it sits (headings), "text": the text to embed}."""
import re

import rag

K = 1   # chunks retrieved per question: this corpus has only ~40 chunks, so one is the fair test


# ---------- Step 1: the measuring stick ----------
def normalise(s):
    return re.sub(r"\s+", " ", s).strip().lower()


def hit(chunk_text, evidence):
    """True when the whole evidence sentence is inside the chunk (ignoring case and line breaks)."""
    # TODO (Step 1): return normalise(evidence) in normalise(chunk_text)
    raise NotImplementedError("Step 1: write hit()")


def evaluate(chunks, questions, k=K):
    """{"recall": share of questions with the evidence in one of the top-k chunks,
        "chars_per_question": average characters of the k chunks sent to the model,
        "chunks": how many chunks, "missed": [question ids with no hit]}"""
    texts = [c["text"] for c in chunks]
    ranked = rag.search(texts, [q["question"] for q in questions], k)
    # TODO (Step 1): for each question and its ranked chunk indices:
    #   add up the characters of the top chunks (chars sent to the model)
    #   count it found if any top chunk hit()s the question's evidence, else add q["id"] to missed
    # Return {"recall": found / len(questions) rounded to 3, "chars_per_question": chars / len(questions) rounded,
    #         "chunks": len(chunks), "missed": missed}
    raise NotImplementedError("Step 1: write evaluate()")


def whole_docs(docs):
    """No chunking at all: each document is one chunk."""
    return [{"doc": name, "path": name, "text": text} for name, text in docs.items()]


# ---------- Step 2: fixed-size chunks ----------
def fixed_chunks(docs, size=400, overlap=0):
    """Cut each document every `size` characters. With overlap, each chunk starts `size - overlap`
    characters after the previous one, so neighbouring chunks share `overlap` characters."""
    raise NotImplementedError("fixed_chunks() arrives in Step 2")


# ---------- Step 3: follow the document's own structure ----------
def section_chunks(docs, max_chars=700):
    """One chunk per section under a markdown heading, with the heading line kept at the top.
    path is the chain of headings, e.g. "Book Box subscription terms > Kids Box > Price".
    A section longer than max_chars is split between paragraphs."""
    raise NotImplementedError("section_chunks() arrives in Step 3")


# ---------- Step 4: give each chunk its context ----------
def with_context(chunks):
    """The same chunks, with the heading path written above each text: "path\\n\\ntext"."""
    raise NotImplementedError("with_context() arrives in Step 4")


# ---------- Step 5: your pick ----------
def best(docs):
    """The chunks you would ship: your best strategy and settings from the shoot-out."""
    raise NotImplementedError("best() arrives in Step 5")
Provided for you:docs/bookbox.mddocs/events.mddocs/shop.mdquestions.jsonrag.pytry_it.py

Frequently asked questions

What is the best chunk size for RAG?

It depends on your documents and questions, which is why the lab measures instead of prescribing. On these documents, chunks that follow headings beat fixed 400-character chunks while sending less text per question.

Does chunk overlap help?

Usually some: in the lab, 100 characters of overlap lifts fixed-size recall by rescuing sentences cut at chunk edges, at the price of storing about a quarter more text.

What is contextual chunking?

Writing where a chunk sits, such as its heading path, into the chunk before embedding it. Sections like 'Price' that repeat across products become distinguishable, and in the lab this is the change that takes recall to the top.

How is recall measured here?

Each question comes with the exact evidence sentence. A question counts as found when the retrieved chunk contains that sentence, ignoring case and line breaks.

Choosing a RAG chunking strategy with measurements

Chunking decides what a RAG system can retrieve, and most advice about it is rules of thumb. The reliable way to choose is to measure: write questions with the exact sentence that answers them, and count how often each strategy retrieves that sentence and how much text it sends. In this lab you build that measurement and use it on real embeddings. You compare fixed-size chunks with and without overlap, chunks that follow the document's headings, and heading-aware chunks with their heading path written in, then confirm the winner on held-out questions.