AST-Based Repository Indexing: Giving an Agent Exact Answers

Grep returns things that look alike. A structural index returns things that are related. Those are different sets, and the gap is where invented answers come from.

Reading files is a poor way to understand a repository

An agent without an index navigates by reading. It greps for a symbol, opens the matches, follows imports, opens more. Each step costs tokens and turns, and the result is an understanding assembled from whichever files happened to match a text search.

An index inverts this. Parse the repository once, build a structural model, and answer questions against the model. A query that would have required reading five files returns one line.

The distinction is not merely efficiency. A text search returns things that look alike; a structural query returns things that are related. Those are different sets, and the difference is where confidently wrong answers come from.

What an AST gives you that grep cannot

The last row is the one that breaks text search outright. In a codebase using interfaces or dependency injection, the relationship between a call site and the code that runs is not textual at all. Grep does not return an incomplete answer there; it returns nothing, and the agent fills the gap by inference.

A minimal index you can build yourself

You do not need a full language server to get most of the benefit. A symbol-to-location map, built from the standard library's own parser, is perhaps forty lines and covers the most common queries.

import ast, json, pathlib

index = {"symbols": {}, "imports": {}}

for path in pathlib.Path("src").rglob("*.py"):
    rel = str(path)
    try:
        tree = ast.parse(path.read_text(encoding="utf-8"))
    except SyntaxError:
        continue

    index["imports"][rel] = []

    for node in ast.walk(tree):
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef,
                             ast.ClassDef)):
            index["symbols"].setdefault(node.name, []).append({
                "file": rel,
                "line": node.lineno,
                "kind": type(node).__name__,
                "doc": (ast.get_docstring(node) or "").split("\n")[0][:120],
            })
        elif isinstance(node, ast.ImportFrom) and node.module:
            index["imports"][rel].append(node.module)
        elif isinstance(node, ast.Import):
            index["imports"][rel] += [a.name for a in node.names]

pathlib.Path(".index.json").write_text(json.dumps(index, indent=1))
print(f"{len(index['symbols'])} symbols across "
      f"{len(index['imports'])} files")

Run it in a pre-commit hook or a watch loop. On a mid-sized repository it takes under a second, and it turns "where is process_payment" from a multi-file read into a dictionary lookup.

Exposing it to the agent

An index nobody queries is a file. The value comes from wiring it to a command the agent knows about.

#!/usr/bin/env python3
# sym — symbol lookup.  Usage: ./sym process_payment
import json, sys, pathlib

idx = json.loads(pathlib.Path(".index.json").read_text())
name = sys.argv[1]

hits = idx["symbols"].get(name)
if not hits:
    near = [k for k in idx["symbols"] if name.lower() in k.lower()][:5]
    print(f"not found: {name}")
    if near:
        print("did you mean: " + ", ".join(near))
    sys.exit(1)

for h in hits:
    print(f"{h['file']}:{h['line']}  {h['kind']}  {h['doc']}")

## Finding code
- `./sym <name>` — where a function or class is defined.
  Use this INSTEAD of grep for symbol lookup. It is exact
  and does not match comments or strings.
- `./sym` prints "not found" when a symbol does not exist.
  Trust that. Do not assume it exists somewhere unindexed.

That last instruction is the important one. The value of an exact index is that "not found" becomes trustworthy — a real signal rather than a gap to be filled by guessing. Say so explicitly, or the agent will treat a negative result as a failure of the tool.

Queries worth exposing beyond symbol lookup

• Reverse imports. "What depends on this module?" is the question before any refactor, and the index already holds the data.
• Layer violations. Walk the import map against your architecture rules and print offenders — a poor man's import linter for free.
• Public surface. Everything a module exports, with signatures. Far cheaper than reading the file when the agent needs the shape rather than the implementation.
• Orphans. Symbols nothing imports. Useful for cleanup, and a good hint that something was superseded.

Language-agnostic indexing

The AST approach above is Python-specific because it uses the standard library's parser. For a polyglot repository, tree-sitter grammars give you the same structural queries across most languages with one interface.

from tree_sitter import Language, Parser

QUERIES = {
    "python": "(function_definition name: (identifier) @name)",
    "go":     "(function_declaration name: (identifier) @name)",
    "ts":     "(function_declaration name: (identifier) @name)",
}

def symbols(path, language, parser, query_src):
    tree = parser.parse(path.read_bytes())
    query = language.query(query_src)
    return [(n.text.decode(), n.start_point[0] + 1)
            for n, _cap in query.captures(tree.root_node)]

The trade is setup cost against coverage. A single-language repository is better served by that language's own parser, which understands more of the semantics. A monorepo spanning four languages is better served by one mechanism that works adequately for all of them than by four that each work well.

Cost, honestly

Worth stating what this does and does not buy, because indexing is easy to oversell.

Row two is worth building when row three is unavailable, and only then. A hand-rolled index resolves names and imports; it does not resolve types, generics, or dynamic dispatch. It is a substantial improvement on grep and a clear step down from a real language server, and pretending otherwise leads teams to build one while a working LSP sits unconfigured in their editor.

Staleness is the whole risk

An index that lags the code is worse than no index, because it answers confidently and wrongly — the same failure this series keeps warning about with stale instructions and stale notes.

Three defences, cheapest first. Rebuild in a pre-commit hook, so it can never be more than one commit behind. Store the source file's modification time with each entry and warn when they disagree. And gitignore the index — it is derived data, and a committed index is one that will be stale in someone else's checkout the moment they pull.

- repo: local
  hooks:
    - id: rebuild-symbol-index
      name: rebuild symbol index
      entry: python3 build_index.py
      language: system
      pass_filenames: false
      always_run: true