Build a RAG offline eval set and fail CI on retrieval regressions

TL;DR — Build a versioned JSONL eval set with queries, expected document IDs, and optional answer checks; score your retriever with a deterministic script that computes Recall@k and MRR; then run that script in CI and fail the build if metrics drop below explicit thresholds. The most common fix is to stop evaluating against mutable indexes: snapshot the corpus and document IDs used by the eval set, or your CI will flap. Reading time: ~5 min

Goal

When you finish, your repo will contain a small offline evaluation set for your retrieval-augmented system, a script that scores retrieval quality against a fixed corpus snapshot, and a CI job that fails with exit code 1 when metrics fall below your thresholds.

Prerequisites

• Git repo with your retriever code and a way to run it from the command line
• Python >= 3.11 — check with:

python3 --version

• jq >= 1.6 — check with:

jq --version

• A fixed corpus snapshot with stable document IDs, exported as JSONL; have the file path ready, for example data/corpus.snapshot.jsonl
• 30-100 representative user queries from production, support logs, or test cases, with sensitive data removed
• CI system that can run shell commands on pull requests (GitHub Actions, GitLab CI, Jenkins, etc.)
• A retriever entry point that accepts a query and returns top-k document IDs; if you do not have one, this article gives a minimal adapter script

Steps

Step 1: Create the versioned eval-set directory

Run:

mkdir -p evals/rag
printf '%s
' '{"schema_version":1,"corpus_snapshot":"data/corpus.snapshot.jsonl","k":5}' > evals/rag/config.json
: > evals/rag/evalset.jsonl

Success looks like: evals/rag/config.json and evals/rag/evalset.jsonl both exist in Git.

Step 2: Add eval examples with expected document IDs

Populate evals/rag/evalset.jsonl with one JSON object per line. Use stable IDs from the corpus snapshot, not titles or URLs.

{"id":"q-001","query":"How do I rotate an API key without downtime?","expected_doc_ids":["doc-1842","doc-7711"],"answer_must_contain":["create new key","revoke old key"]}
{"id":"q-002","query":"What is the retention period for audit logs?","expected_doc_ids":["doc-220","doc-221"],"answer_must_contain":["90 days"]}
{"id":"q-003","query":"How do I reset MFA for a locked-out user?","expected_doc_ids":["doc-991"],"answer_must_contain":["identity verification"]}

Success looks like: wc -l evals/rag/evalset.jsonl returns the number of test queries you added.

Step 3: Add a retriever adapter that returns top-k document IDs

Create scripts/retrieve.py and adapt the search() function to your codebase. The interface is fixed: input query and --k, output JSON array of document IDs to stdout.

import argparse, json, sys

# Replace this import with your actual retriever implementation.
# from app.retrieval import search

def search(query: str, k: int) -> list[str]:
    raise NotImplementedError("Replace search() with your retriever call that returns stable document IDs")

if __name__ == "__main__":
    p = argparse.ArgumentParser()
    p.add_argument("query")
    p.add_argument("--k", type=int, default=5)
    args = p.parse_args()
    ids = search(args.query, args.k)
    json.dump(ids, sys.stdout)

Success looks like: running the command below prints a JSON array such as ["doc-1842","doc-7711","doc-88"].

python3 scripts/retrieve.py "How do I rotate an API key without downtime?" --k 5

Step 4: Add the scoring script

Create scripts/score_rag_eval.py:

import argparse, json, subprocess, sys
from pathlib import Path

def recall_at_k(expected, got):
    return 1.0 if set(expected) & set(got) else 0.0

def mrr(expected, got):
    expected = set(expected)
    for i, doc_id in enumerate(got, start=1):
        if doc_id in expected:
            return 1.0 / i
    return 0.0

def run_retriever(query, k):
    p = subprocess.run([sys.executable, "scripts/retrieve.py", query, "--k", str(k)], capture_output=True, text=True)
    if p.returncode != 0:
        print(p.stderr, file=sys.stderr)
        raise SystemExit(2)
    return json.loads(p.stdout)

if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("--evalset", default="evals/rag/evalset.jsonl")
    ap.add_argument("--k", type=int, default=5)
    ap.add_argument("--min-recall", type=float, default=0.80)
    ap.add_argument("--min-mrr", type=float, default=0.60)
    args = ap.parse_args()

    rows = [json.loads(line) for line in Path(args.evalset).read_text().splitlines() if line.strip()]
    total_recall = 0.0
    total_mrr = 0.0
    failures = []

    for row in rows:
        got = run_retriever(row["query"], args.k)
        r = recall_at_k(row["expected_doc_ids"], got)
        rr = mrr(row["expected_doc_ids"], got)
        total_recall += r
        total_mrr += rr
        if r == 0.0:
            failures.append({"id": row["id"], "query": row["query"], "expected": row["expected_doc_ids"], "got": got})

    n = max(len(rows), 1)
    avg_recall = total_recall / n
    avg_mrr = total_mrr / n
    print(json.dumps({"queries": n, "k": args.k, "recall_at_k": round(avg_recall, 4), "mrr": round(avg_mrr, 4), "failed_queries": failures[:10]}, indent=2))

    if avg_recall < args.min_recall or avg_mrr < args.min_mrr:
        raise SystemExit(1)

Success looks like: the script prints JSON metrics and exits 0 when thresholds pass.

Step 5: Run the eval locally and inspect failures

Run:

python3 scripts/score_rag_eval.py --evalset evals/rag/evalset.jsonl --k 5 --min-recall 0.80 --min-mrr 0.60
printf 'exit_code=%s
' "$?"

Success looks like output in this shape:

{
  "queries": 35,
  "k": 5,
  "recall_at_k": 0.8571,
  "mrr": 0.6762,
  "failed_queries": [
    {
      "id": "q-014",
      "query": "Can I export audit logs to S3?",
      "expected": ["doc-552"],
      "got": ["doc-77", "doc-81", "doc-552", "doc-90", "doc-91"]
    }
  ]
}

If thresholds fail, the last line should be exit_code=1.

Step 6: Pin the corpus snapshot used by the eval

⚠️ If your retriever reads a live index or mutable database, CI results will drift and fail unrelated pull requests. Switch the eval path to a read-only snapshot before you rely on the exit code.

Commit the snapshot manifest and, if practical, the snapshot file checksum:

sha256sum data/corpus.snapshot.jsonl > data/corpus.snapshot.jsonl.sha256
git add evals/rag/config.json evals/rag/evalset.jsonl data/corpus.snapshot.jsonl.sha256 scripts/retrieve.py scripts/score_rag_eval.py
git commit -m "Add offline RAG eval set and scorer"

Success looks like: git show --stat --oneline HEAD includes the eval files and checksum.

Step 7: Add the CI job that fails on regression

For GitHub Actions, create .github/workflows/rag-eval.yml:

name: rag-eval
on:
  pull_request:
  push:
    branches: [ main ]
jobs:
  offline-rag-eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Install deps
        run: |
          python -m pip install --upgrade pip
          if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
      - name: Verify corpus checksum
        run: sha256sum -c data/corpus.snapshot.jsonl.sha256
      - name: Score retrieval
        run: python scripts/score_rag_eval.py --evalset evals/rag/evalset.jsonl --k 5 --min-recall 0.80 --min-mrr 0.60

Success looks like: the CI job is green on a good branch and red on a branch that regresses retrieval.

Verify it works

Run locally:

python3 scripts/score_rag_eval.py --evalset evals/rag/evalset.jsonl --k 5 --min-recall 0.80 --min-mrr 0.60
echo $?

Expected result:

0

Then push a deliberate regression, for example return an empty list from search(), and rerun CI. Expected failure shape:

{
  "queries": 35,
  "k": 5,
  "recall_at_k": 0.0,
  "mrr": 0.0,
  "failed_queries": [
    {
      "id": "q-001",
      "query": "How do I rotate an API key without downtime?",
      "expected": ["doc-1842", "doc-7711"],
      "got": []
    }
  ]
}

And the CI step should end with exit code 1.

Common pitfalls

Mutable document IDs

Mistake: using titles, URLs, or database row numbers that change between environments as expected_doc_ids. Symptom: local runs pass, CI fails with many misses even though the right content exists. Fix: export stable IDs into data/corpus.snapshot.jsonl and use only those IDs in evals/rag/evalset.jsonl.

Evaluating against a live index

Mistake: CI hits the current production or staging index instead of a snapshot. Symptom: the same commit flips between pass and fail; failed queries differ on every run. Fix: point scripts/retrieve.py at a read-only snapshot or a test index rebuilt from data/corpus.snapshot.jsonl.

Thresholds copied from a larger system

Mistake: setting --min-recall 0.95 --min-mrr 0.90 on a 20-query eval set. Symptom: CI blocks nearly every retrieval change, including harmless ranking tweaks. Fix: start with your current baseline plus a small guardrail, for example current 0.84/0.65 becomes 0.82/0.62, then tighten after the eval set grows.

Query set polluted with duplicates

Mistake: adding the same query phrased slightly differently 10 times from one incident. Symptom: metrics look great on one topic and miss regressions elsewhere. Fix: deduplicate by intent before committing; keep coverage across top user tasks, failure modes, and rare but important queries.

Mixing retrieval and generation failures

Mistake: failing the retrieval CI gate because the final answer text changed, even when the right docs were retrieved. Symptom: failed_queries show expected docs present in got, but another test still fails. Fix: gate CI on retrieval metrics here; keep answer-quality checks in a separate job with different thresholds and owners.

Hidden dependency drift in CI

Mistake: retriever behavior changes because embeddings, tokenizers, or ranking libraries are not pinned. Symptom: checksum passes, code unchanged, but metrics shift after dependency updates. Fix: pin dependency versions in requirements.txt or lockfiles and install them in CI before scoring.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.