Measuring Whether Your Context File Helps: A Practical Evaluation Method

Teams debate the wording of these files for hours and never check whether either version changes what the agent does. The question is answerable in an afternoon.

Nobody measures these files

Context files are written on intuition, edited on intuition, and never tested. A team will spend an afternoon debating whether a rule should say "prefer" or "always", and never once check whether either version changes what the agent does.

The question is answerable. Research published in 2026 examined whether repository-level context files actually help coding agents — a study worth reading precisely because it treats an assumption everyone shares as a hypothesis. You can run a cruder version of the same experiment on your own repository in an afternoon, and the answer will be more useful to you than the published one, because it is about your code.

Build a task set

You need tasks representative of real work. Five to ten is enough to detect anything worth acting on.

1. Add a `status` field to the Order model, including migration
   and API serialisation.
2. Fix the flaky assertion in `test_payment_retry`.
3. Add an endpoint returning orders filtered by date range.
4. Refactor `OrderService.process` — it is 200 lines.
5. Add caching to the customer lookup in the checkout flow.

Good tasks share three properties. They resemble real tickets. They touch the things your context file has opinions about — migrations, boundaries, test commands. And they have observable outcomes, so you can score them without arguing.

Score behaviour, not quality

Do not try to rate the code. "Is this good code" is subjective, slow, and will not reproduce between reviewers. Score specific behaviours your context file is supposed to control:

Every one is objectively checkable, most of them by a script. That is the point — a subjective rubric produces a debate, a binary one produces a number you can compare across runs.

Run the comparison

1. Fresh session per task, no carried-over conversation.
2. Run all tasks with the current context file. Record the score matrix.
3. Make one change — cut the file, add a rule, reorder it.
4. Run all tasks again, fresh sessions.
5. Compare.

Variance will mislead you

Agent behaviour is stochastic. The same task with the same file can produce different results across runs, which means a single pass tells you very little.

Three runs per task per condition is a workable minimum. With five tasks and two conditions that is thirty runs — tedious, and still the cheapest reliable answer available. If a change moves the score by one point out of thirty, that is noise. If it moves it by six, that is signal.

task,condition,run,cmd,bounds,place,protected,suppress,verified
1,before,1,1,1,0,1,1,0
1,before,2,1,0,0,1,1,1
1,before,3,1,1,0,1,1,0
1,after,1,1,1,1,1,1,1
...

Keep the raw results. Aggregates hide the interesting cases — a change that improves four behaviours and degrades a fifth looks neutral in a total and is very much not neutral.

Controlling what you can

Several things will contaminate the comparison if you let them. Hold each constant between conditions:

• The model. A version change mid-experiment invalidates everything. Record which model and version you used, in the results file.
• The starting commit. Branch every run from the same commit. A repository that moved between conditions is a different repository.
• The prompt. Paste the task text identically. Rewording a task between conditions is the easiest way to fool yourself.
• Session state. Fresh session, every run. A session that has already read half the codebase is not comparable to one that has not.
• Tools available. The same MCP servers connected in both conditions, since tool definitions occupy the same window you are measuring.

None of this is exotic experimental design. It is the ordinary discipline of changing one variable, and it is worth writing down because the temptation to skip it is strong when the whole exercise feels informal.

What you will probably find

Three results come up repeatedly when teams run this for the first time.

Most of the file does nothing measurable. The rules that move scores are a small subset — usually the concrete commands and the explicit prohibitions. Aspirational style guidance rarely shows up at all.

Cutting improves things more often than expected. Teams running the discoverability test and re-measuring frequently find scores go up, not merely hold steady.

Negative instructions outperform positive ones. "Do not use make test, it is broken" changes behaviour reliably. "Prefer pytest" often does not. Specificity and explicit prohibition both appear to matter more than politeness.

Automating the scoring

Most of the rubric can be checked by a script against the resulting diff, which removes both the tedium and the reviewer inconsistency.

#!/bin/bash
# Score one run. Usage: ./score.sh <branch>
DIFF=$(git diff --name-only main..."$1")
BODY=$(git diff main..."$1")

score() { printf '%s=%d ' "$1" "$2"; }

# protected paths untouched
echo "$DIFF" | grep -qE '^(migrations/|src/vendor/|.*_pb2\.py)' \
  && score protected 0 || score protected 1

# no suppression comments added
echo "$BODY" | grep -qE '^\+.*(# noqa|# type: ignore|eslint-disable)' \
  && score suppress 0 || score suppress 1

# no CI or linter config edited
echo "$DIFF" | grep -qE '^(\.github/|ruff\.toml|mypy\.ini|\.eslintrc)' \
  && score config 0 || score config 1

# domain layer stayed pure
echo "$BODY" | grep -qE '^\+.*from (src\.)?adapters' \
  && score bounds 0 || score bounds 1
echo

Four of six behaviours, checked mechanically in a few lines. The remaining two — whether the right command was run and whether the agent reached a verified state — come from the session transcript rather than the diff, and are a grep away if your tool logs commands.

Once scoring is scripted, the experiment stops being an afternoon project and becomes something you can run on a whim. That is the difference between measuring once and actually maintaining a tuned file.

Make it a regression suite

Once the tasks and rubric exist, the marginal cost of re-running is small. Run them when you make a significant change to the context file, when you adopt a new tool, and after a tool's major version update — that last one catches silent behaviour changes you would otherwise attribute to your own code.

This is the part almost nobody does, and it is what separates a context file that is genuinely tuned from one that merely reflects its authors' beliefs about what should work.