Preporato
Hybrid Search and Reranking: BM25, Embeddings, RRF and an LLM Reranker
Hosted · ide
Beta

Hybrid Search and Reranking: BM25, Embeddings, RRF and an LLM Reranker

Build hybrid retrieval step by step and measure every stage: BM25 keyword search, dense embedding search, reciprocal rank fusion, and a listwise LLM reranker that orders the candidate pool. Size the rerank pool from a recall and prompt-length report and prove the pipeline on questions it has never seen.

55 min5 steps3 domainsIntermediate

Hands-on labs require Pro · $29.99/mo · cancel anytime

What you'll learn

  1. 1
    Keyword search with BM25
    Hearthly's help centre has 56 short articles (kb.json). questions.json holds 43 real-looking customer
  2. 2
    Dense search with embeddings
    BM25 found most exact-detail questions and missed half of the paraphrased ones: "I got a new broadband box"
  3. 3
    Fuse the rankings with RRF
    Each search finds answers the other misses. You cannot add their scores: BM25 scores run from 0 to about 20,
  4. 4
    Rerank the pool with a model
    Hybrid search almost always has the answer in its top ten, but it is often not first. BM25 and embeddings
  5. 5
    Size the pool and ship
    The shipped pipeline, answer_passages(), reranks the top SETTINGS["rerank_top"] hybrid results. That

Step 1, as you will see it

This is the lab’s own text. Each step ends with a check that runs your work in the lab environment; the hint and the solution stay inside the lab.

Step 1: Keyword search with BM25

Hearthly's help centre has 56 short articles (kb.json). questions.json holds 43 real-looking customer questions, each with the id of the article that answers it, split into three kinds:

  • exact: the question names a detail from the article: an error code, a model number, a terminal, a figure.
  • paraphrase: the customer describes the problem in their own words.
  • plain: ordinary questions that share words with their answer.

evaluate.py scores any search on them: hit@1 is how often the right article comes first, and recall@5 is how often it is anywhere in the top five.

BM25 is the classic keyword ranking: an article scores for every question word it contains, more for rare words, less when the article is long. It only ever sees tokens, so how you cut text into tokens decides what can match: "E41," and "e41" must become the same token.

Do this

1. Write tokenize(text): lowercase the text and return its runs of letters and digits. "Error E41, HT-2000!" becomes ["error", "e41", "ht", "2000"].

2. Write bm25_search(question, k): build BM25Okapi from the tokenized TEXTS once (keep it in _bm25), score the tokenized question with get_scores(), and return the ids of the k best articles.

3. Run. Compare the exact and paraphrase rows before you check.

Starter file: evaluate.py

"""Scores search functions on questions.json. Run it with the method to score:
    python3 evaluate.py bm25        python3 evaluate.py dense        python3 evaluate.py hybrid
    python3 evaluate.py rerank      python3 evaluate.py shipped      python3 evaluate.py pool
hit@1 = the right passage is first; recall@5 = it is anywhere in the top five. Scores are split by kind of
question: exact (error codes, model numbers, versions), paraphrase (the customer's own words), plain."""
import json
import sys
import time
from concurrent.futures import ThreadPoolExecutor

import search

KINDS = ["exact", "paraphrase", "plain"]


def ranked(method, question):
    if method == "bm25":
        return search.bm25_search(question, 5)
    if method == "dense":
        return search.dense_search(question, 5)
    if method == "hybrid":
        return search.hybrid_search(question, 5)
    if method == "rerank":
        return search.rerank(question, search.hybrid_search(question, 10))[:5]
    if method == "shipped":
        return search.answer_passages(question, 5)
    raise SystemExit(f"unknown method {method!r}: use bm25, dense, hybrid, rerank or shipped")


def score(method, questions):
    rows = []
    with ThreadPoolExecutor(6) as pool:  # rerank calls the model once per question; six at a time
        results = list(pool.map(lambda q: ranked(method, q["question"]), questions))
    for q, ids in zip(questions, results):
        rows.append({"id": q["id"], "kind": q["kind"], "top": ids[:5],
                     "hit1": bool(ids) and ids[0] == q["answer"], "hit5": q["answer"] in ids[:5]})
    out = {}
    for kind in KINDS + ["all"]:
        sel = [r for r in rows if kind == "all" or r["kind"] == kind]
        out[kind] = {"n": len(sel), "hit@1": round(sum(r["hit1"] for r in sel) / len(sel), 3),
                     "recall@5": round(sum(r["hit5"] for r in sel) / len(sel), 3)}
    return out, rows


def pool_table(questions):
    print(f"rerank pool sizes (hybrid depth {search.SETTINGS['depth']})")
    print(f"  {'size':>4}{'recall':>8}{'avg prompt chars':>18}")
    for r in search.pool_report(questions, range(4, 21), search.SETTINGS["depth"]):
        print(f"  {r['size']:>4}{r['recall']:>8.3f}{r['avg_prompt_chars']:>18}")
    print(f"  shipping rerank_top = {search.SETTINGS['rerank_top']}")


if __name__ == "__main__":
    method = sys.argv[1] if len(sys.argv) > 1 else "bm25"
    questions = json.load(open("questions.json", encoding="utf-8"))
    if method == "pool":
        pool_table(questions)
        raise SystemExit
    t = time.time()
    table, rows = score(method, questions)
    print(f"{method}  ({len(questions)} questions, {time.time() - t:.1f}s)")
    print(f"  {'kind':<11}{'n':>4}{'hit@1':>8}{'recall@5':>10}")
    for kind, s in table.items():
        print(f"  {kind:<11}{s['n']:>4}{s['hit@1']:>8.2f}{s['recall@5']:>10.2f}")
    miss = [r for r in rows if not r["hit5"]]
    if miss:
        print("  not in top 5:", ", ".join(f"q{r['id']}" for r in miss))
    json.dump({"method": method, "table": table, "rows": rows}, open(f".last_{method}.json", "w"), indent=1)

Prerequisites

  • Python: functions, lists, dictionaries
  • Helpful: Embeddings You Can See, RAG Chunking Shoot-Out

Exam domains covered

RAGSearchEvaluation

Skills & technologies you'll practice

This intermediate-level ai/ml lab gives you real-world reps across:

RAGhybrid searchBM25rerankingreciprocal rank fusionintermediate

Hybrid search with reranking, measured stage by stage

Hybrid search combines keyword ranking (BM25) with embedding search, and a reranker then orders the combined candidates. Each stage fixes a different failure, and the way to see that is to measure every stage on the same labelled questions. In this lab you write BM25 search, dense search, reciprocal rank fusion and a listwise reranker that asks a chat model to order a numbered pool, with a parser that never loses a candidate. You then size the rerank pool from a recall and prompt-length report and check the result on held-out questions.

Frequently asked questions

What is hybrid search?

Running keyword search (such as BM25) and embedding search on the same query and merging the two rankings. Keyword search matches exact details like error codes and figures; embeddings match questions written in different words.

What is reciprocal rank fusion?

A way to merge rankings using positions only: each list gives a result 1 / (k + rank), usually with k = 60, and the totals set the final order. It needs no score normalisation, because BM25 scores and cosine similarities are on different scales.

What does a reranker add?

It reads the question together with each candidate and orders them by how well they answer it. In the lab, hybrid search has the answer in its top ten for every question but first for only about three in four; the reranker puts it first almost every time.

How many candidates should a reranker see?

Enough that the answer is almost always in the pool, and no more, because every candidate lengthens the prompt. The lab measures recall and prompt length for pool sizes 4 to 20 and picks the smallest size that reaches full recall.