RAG for Runbooks: The Narrow Case Where Internal Retrieval Works

An index over all your documentation faithfully surfaces everything stale and contradictory in it. An index over forty current runbooks answers the question someone has at 3am.

Runbooks are the case where retrieval earns its keep

Most internal-documentation RAG projects disappoint, because most internal documentation is stale, contradictory and duplicated, and retrieval faithfully surfaces all three properties. The exception is operational content — runbooks, incident write-ups, on-call procedures — where the material is specific, consulted under time pressure, and painful to search by hand at 3am.

That is a narrow enough target to build well, and narrow is the whole trick. An index over "all our documentation" produces confident answers assembled from a 2023 wiki page and a superseded design doc. An index over forty current runbooks produces answers you can act on.

Curate ruthlessly, then index

The superseded-ADR row is deliberate. An architecture decision that was reversed is valuable context — it tells you the approach was tried — provided the record says so. Index it with its status in the text, not only in metadata, because the status has to survive into the retrieved chunk.

Chunk on structure, not on length

The default of splitting every 500 tokens with an overlap is what makes retrieval feel unreliable. It cuts procedures in half, separates a command from the warning above it, and produces chunks that begin mid-sentence.

import re, pathlib, json

def chunk_markdown(path: pathlib.Path):
    text = path.read_text(encoding="utf-8")
    # Split on H2 boundaries — in a runbook these are the procedures.
    parts = re.split(r'\n(?=## )', text)
    header = parts[0] if not parts[0].startswith("## ") else ""

    for part in parts:
        title = (re.match(r'## (.+)', part) or [None, "overview"])[1]
        yield {
            "id": f"{path.stem}#{title}",
            "source": str(path),
            "section": title,
            # Prepend the document title so a retrieved chunk is
            # self-describing. Without this, "restart the worker"
            # arrives with no indication of which service.
            "text": f"# {path.stem}\n## {title}\n{part}",
            "owner": extract_owner(header),
            "updated": extract_date(header),
        }

The prepended title is the single highest-value detail. A retrieved chunk reading "restart the worker, then check the queue depth" is dangerous without knowing which service it belongs to, and fixed-length chunking loses that routinely.

Return provenance, always

A RAG answer without sources is worse than a search result, because it looks authoritative. For operational content that is a real hazard — someone follows a procedure at 3am that was correct eighteen months ago.

{
  "answer": "Restart the ingestion workers with `kubectl rollout restart deploy/ingest -n data`, then confirm queue depth drops in the Kafka dashboard.",
  "sources": [
    {"doc": "runbooks/ingest-backlog.md",
     "section": "Clearing a backlog",
     "owner": "@data-platform",
     "updated": "2026-04-02",
     "age_days": 55}
  ],
  "warning": null
}

Include the age, and set a threshold that produces a warning rather than silence. A procedure last touched fourteen months ago should arrive with that fact attached, because the reader's next question is exactly "is this still true".

Hybrid retrieval, because exact terms matter

Pure vector search is poor at the queries operational staff actually type. Error codes, service names, metric names and command flags are exactly the tokens where lexical matching wins and embeddings blur.

def retrieve(query: str, k: int = 6):
    dense  = vector_index.search(embed(query), k=20)
    sparse = bm25_index.search(query, k=20)

    # Reciprocal rank fusion — no score normalisation needed.
    scores = {}
    for rank, hit in enumerate(dense):
        scores[hit.id] = scores.get(hit.id, 0) + 1 / (60 + rank)
    for rank, hit in enumerate(sparse):
        scores[hit.id] = scores.get(hit.id, 0) + 1 / (60 + rank)

    top = sorted(scores, key=scores.get, reverse=True)[:k]
    return [store[i] for i in top]

Twelve lines, and it fixes the most common complaint about internal RAG — that searching for the literal error string in your logs returns thematically-related documents instead of the runbook that names it.

Wire it as a tool, not as a context dump

The temptation is to retrieve on every turn and prepend the results. Better to expose retrieval as a tool the agent calls when it decides documentation would help.

{
  "name": "search_runbooks",
  "description":
    "Search internal runbooks, incident write-ups and ADRs. Use when a "
    "question concerns how WE operate a specific service — deployment, "
    "on-call procedures, known failure modes. Returns passages with "
    "source, owner and last-updated date. Do not use for general "
    "technical questions; this only contains our own operational docs."
}

The final sentence prevents the common failure of a retrieval tool being called for everything and returning irrelevant passages that then pollute the answer.

Keeping the index fresh

Documentation retrieval decays faster than code retrieval, because nothing breaks when a runbook goes stale. Two mechanisms keep it honest, and both are cheap.

Rebuild on merge, not on a schedule. A cron job that reindexes nightly means a corrected runbook is wrong for up to a day — which is exactly the day someone needs it. Trigger the rebuild from the documentation repository's CI.

Surface the ownerless. A weekly report of indexed documents with no owner or no edit in twelve months, sent to the team that owns the service, is the only mechanism that reliably gets runbooks updated. Nobody reviews documentation voluntarily; they respond to a list with their name on it.

#!/bin/bash
# Weekly: runbooks nobody has touched in a year
find runbooks -name '*.md' -mtime +365 | while read -r f; do
  owner=$(grep -m1 '^owner:' "$f" | cut -d' ' -f2)
  printf '%-45s %-18s %s
' "$f" "${owner:-UNOWNED}"          "$(date -r "$f" +%Y-%m-%d)"
done | sort -k2

Evaluate it, or it will quietly rot

Write thirty real questions — taken from your incident channel, not invented — with the document that should be retrieved for each. Then measure one thing: is the right document in the top five?

{"q": "ingest queue is backing up, what do I do",
 "expect": "runbooks/ingest-backlog.md"}
{"q": "error 5023 from the payments gateway",
 "expect": "runbooks/payments-gateway-errors.md"}
{"q": "who do I page for auth service at 2am",
 "expect": "oncall/escalation.md"}

Recall@5 is the metric that predicts whether people keep using the system. Answer quality is downstream of it: if the right document is not retrieved, no amount of prompting fixes the answer. Re-run the set whenever you change chunking, the embedding model, or the retrieval mix.