Writing Build and Test Instructions an Agent Can Actually Execute

An agent that can run your tests catches its own mistakes. An agent that can't hands you plausible code it never verified. This is the highest-leverage section of the file.

Why the test command is the highest-value line in the file

An agent that cannot run your tests cannot verify its own work. It writes code, cannot check it, and hands you something plausible. An agent that can run your tests catches its own mistakes before you see them, and the difference in output quality is larger than any other single thing you can put in a context file.

Which makes it worth being precise about, because "run the tests" hides a surprising amount of complexity.

Five ways the obvious instruction fails

The command is aspirational

make test is in the README. It broke eight months ago when someone changed the Docker setup. Everyone on the team knows to run pytest directly. Nobody updated the README because nobody reads it. The agent reads it, runs a broken command, sees an error, and concludes its own code is at fault.

It needs setup nobody mentions

The integration tests need a database. Locally, everyone has one running from a docker compose up -d they did on their first day and never thought about again. The agent does not have one, hangs for the connection timeout, and reports failures unrelated to anything it wrote.

It is too slow to be useful

The full suite takes fourteen minutes. An agent that runs it after every change spends its session waiting. Worse, some agents will decide the run has hung and kill it. You need to tell it which subset to run during iteration and which to run before finishing.

The failure output is misleading

Some suites emit warnings that look like failures, or exit non-zero on conditions the team ignores. If your test run prints a wall of deprecation warnings and exits 1 because of a flaky snapshot, say so explicitly.

It is not the check that actually gates merging

If CI also runs a type checker, a linter and a schema validation, and any of them can block the PR, then "the tests pass" is not the completion criterion. The agent should know the whole gate, not one part of it.

Writing it so it survives contact

## Commands

Fast loop — run after every change:
  `pytest tests/unit -x -q`          (~8s, no external deps)

Before you finish — all four must pass:
  `pytest tests/unit -q`
  `pytest tests/integration -q`      needs `docker compose up -d` first
  `mypy src/`                        zero errors, no `# type: ignore` added
  `ruff check .`

Notes:
- `make test` is BROKEN. Do not use it. Use pytest directly.
- Integration tests print ~40 lines of urllib3 warnings. Ignore them;
  only the final summary line matters.
- If integration tests hang for 60s then fail, Docker is not running.

Notice what this does. It separates the fast inner loop from the full gate, so the agent is not waiting on a fourteen-minute suite between edits. It names the completion criterion explicitly — four commands, all passing. It pre-empts the two failure modes that would otherwise send the agent chasing imaginary bugs. And it explicitly contradicts a stale instruction that exists elsewhere in the repo.

Build commands have the same problems

Everything above applies to builds, with one addition: builds frequently have modes, and the agent will pick the wrong one. A production build that takes four minutes when a dev build takes nine seconds is a real cost across a session.

## Build
- Dev (use this): `npm run dev` — incremental, ~2s
- Production check (only before finishing): `npm run build`
- Do NOT run `npm run build:analyze` — it opens a browser window

That last line prevents a specific and irritating failure: a command that blocks waiting for something the agent cannot provide.

State the completion criterion explicitly

The single most useful sentence in this whole section is the one that defines done. Without it, the agent invents a criterion, and its invented criterion is usually "the code looks right".

Work is complete when pytest -q, mypy src/ and ruff check . all pass with zero errors, and no new # type: ignore or # noqa comments have been added.

That last clause matters more than it looks. Told to make the type checker pass, a sufficiently determined agent will make the type checker pass — by suppressing the errors. Naming the suppression mechanisms closes the loophole.

Telling the agent how to read a failure

A test failure is a signal that needs interpreting, and the interpretation is not always obvious from the output. Three categories are worth distinguishing explicitly, because the correct response differs for each.

The middle row is the dangerous one. An agent told to make the tests pass, faced with a test asserting behaviour its change deliberately altered, will often edit the assertion. Sometimes that is right. Frequently the test was encoding a requirement nobody mentioned, and quietly rewriting it deletes the requirement.

## When tests fail
- Do NOT modify a test to make it pass unless the task explicitly
  asked you to change that behaviour. If a test asserts something
  your change contradicts, stop and say so.
- Tests in `tests/contract/` encode external API guarantees.
  Never modify these. A failure there means the change is wrong.
- Flaky: `test_websocket_reconnect` fails ~1 in 10 for timing.
  Re-run once before investigating.

The last line prevents a specific waste: an agent spending twenty minutes debugging a known-flaky test. If you have flaky tests, name them. The honest version of this instruction is uncomfortable to write down, which is part of why it is valuable — writing it down occasionally shames a team into fixing the flake.

Make the commands runnable, not just documented

The most durable version of this advice: wherever possible, put the real command behind a single stable entry point that you actually maintain, and point the context file at that.

## Commands
- Everything: `./check.sh` — runs unit, types and lint in order,
  stops at the first failure. This is exactly what CI runs.

Three lines instead of fifteen, and it cannot drift, because if check.sh breaks, CI breaks and someone fixes it that day. You have moved the instruction from prose — which rots silently — into executable code, which does not.