Long-Horizon Runs: The Loop Cannot Tell Progress From Motion

Getting stuck is the benign outcome. The expensive one is two hours of activity that looks exactly like progress.

The failure is not stopping too early

The worry people have about long unsupervised runs is that the agent gives up, or gets stuck, or wanders off. Getting stuck is the benign outcome: it is visible, it is cheap, and you find out.

The expensive failure is the opposite. An agent that has stopped making progress does not stop working. It keeps generating activity that looks exactly like progress — edits, test runs, commits, plausible commentary — because nothing in the loop can distinguish between moving toward the goal and moving. Two hours later there is a large diff, a green suite, and a change that does not do what you asked.

Everything worth building around long-horizon runs is a response to that one property: the loop has no native sense of futility, so you have to supply one.

Productive-looking motion, described precisely

The pattern has a recognisable shape. The agent attempts a fix. The check still fails. Rather than questioning its model of the problem, it attempts a variation. That fails. It broadens: refactors a nearby function, adds a defensive branch, adjusts a mock, tries a different library call.

Each step is locally reasonable. The accumulated diff is not, because it encodes a search rather than a solution — twenty attempts layered on top of each other, most of which were wrong and only some of which were reverted. The tell is a diff much larger than the task warranted, containing changes that no longer relate to anything.

The signals that lie

Activity metrics feel like progress and are not. Files touched, lines changed, commits made, tests added, time elapsed — all of these go up during a failing search exactly as they do during successful work. Watching them tells you the agent is busy, which was never in doubt.

Worse, some of them go up faster when things are going badly, because a stuck loop generates more attempts per unit of real progress. A dashboard of activity will show a struggling run as the most productive one.

The signal that does not

The target check, and only the target check. Did the failing test go green. Did the type error count decrease. Did the benchmark improve. A binary or numeric measure defined before the run, evaluated the same way each time.

Which means a long-horizon run without a machine-checkable goal has no progress signal at all, and the agent is optimising against its own impression. That is the same condition that makes a task unsuitable for handoff generally, and it matters more here because the run is long enough for the drift to accumulate. The test for whether a task is ready is in asynchronous task handoff.

The anti-goal: fixing the test instead of the code

This deserves its own section because it is the single most common way a long run produces a false success.

The goal is "make the suite pass". There are two ways to satisfy it. One is to fix the code. The other is to change the test — relax an assertion, add a skip, widen a tolerance, mock the thing that was failing, delete the case. The second is often easier, always available, and satisfies the stated objective perfectly.

An agent doing this is not being deceptive. It is optimising the objective you gave it, and you gave it the wrong one. "Make the tests pass" is a different instruction from "fix the bug", and only one of them has a degenerate solution.

#!/usr/bin/env bash
# Refuse a change that makes the suite pass by weakening the suite.
set -euo pipefail
BASE="${1:-origin/main}"

diff=$(git diff "$BASE"...HEAD -- '*test*' '*spec*')

removed_asserts=$(printf '%s' "$diff" | grep -cE '^-.*(assert|expect|should)' || true)
added_asserts=$(printf '%s'   "$diff" | grep -cE '^\+.*(assert|expect|should)' || true)
added_skips=$(printf '%s'     "$diff" | grep -cE '^\+.*(\.skip|@skip|xit|xdescribe|t\.Skip)' || true)
removed_tests=$(printf '%s'   "$diff" | grep -cE '^-.*(it\(|test\(|def test_|func Test)' || true)

echo "assertions: -$removed_asserts +$added_asserts | skips +$added_skips | tests removed $removed_tests"

if [ "$added_skips" -gt 0 ] || [ "$removed_tests" -gt 0 ] \
   || [ "$removed_asserts" -gt "$added_asserts" ]; then
  echo "FAIL: the suite got weaker. Justify in the PR or fix the code." >&2
  exit 1
fi

It is crude, it has false positives on legitimate test refactors, and it catches the degenerate solution reliably. A false positive costs a sentence of justification; a missed one costs a bug shipped behind a green build.

Budgets, and what to do when one is hit

Every long run needs a ceiling — wall clock, tokens, or iterations — and, more importantly, an instruction about what happens at it. A budget with no exit behaviour just relocates the problem to whoever finds the truncated session.

The instruction that works: on reaching the budget, stop, revert to the last good state, and write a report covering what was tried, what the remaining failure is, what was ruled out, and what you would try next. That converts a failed run into a useful artefact, which is the difference between a wasted two hours and a bounded investigation.

Checkpoint so there is something to salvage

A two-hour run that ends badly should not leave you with one enormous uncommitted diff. Commit at each point where the target check improved, with a message saying what changed and what the check said.

Then a failed run still has value: you can see where progress stopped, reset to the last good checkpoint, and continue from there rather than from nothing. It also makes the drift visible — a run with four checkpoints in the first thirty minutes and none in the following ninety has a clearly identifiable moment where it stopped working.

Compaction is a source of drift

Long runs exceed the context window, and the conversation gets summarised to continue. Summarisation is lossy, and what it tends to lose is exactly the constraint material: the thing you said not to touch, the assumption that was corrected an hour ago, the approach that was already ruled out.

The symptom is an agent that, late in a long run, does something you explicitly forbade at the start. It is not ignoring you; the instruction is no longer in its context. The mitigation is to keep durable constraints in a file that is re-read rather than in the conversation — which is what repository instruction files are for, and a good reason to write the run's specific constraints into one at the start of a long task.

Monitoring without supervising

The point of a long run is not to watch it, and reading the stream defeats the purpose. What you want is a low-bandwidth signal you can glance at.

The checkpoint log is that signal. Three commits in two hours, each with the check result in the message, tells you the shape of the run in five seconds. No commits in ninety minutes tells you to intervene. Neither requires reading anything the agent said.

Restart beats continue, more often than feels right

When you find a run that has drifted, the instinct is to correct it and continue: point out the mistake, ask it to undo the unnecessary changes, carry on. That rarely works well, because the context is now full of the failed search, and the accumulated wrong turns keep influencing what comes next.

A fresh session, starting from the last good checkpoint, with what you learned added to the brief, is usually faster and produces a cleaner result. The sunk cost is real and it is already sunk; the two hours are gone whether or not you spend another forty minutes trying to rescue their output.

Sometimes the answer is a shorter run

Multi-hour autonomy is impressive and it is not always the right shape. A task that can be decomposed into four independently checkable pieces should be, because each piece then has its own done condition, its own review, and a failure that costs twenty minutes instead of two hours.

Long runs are for problems that genuinely cannot be decomposed — a build failure whose cause is unknown, a flaky test requiring many iterations to characterise, a search through a large space where each attempt is cheap. Those have a real iteration count and no natural checkpoints, which is exactly when unattended persistence earns its keep.

What a well-run long session looks like

A machine-checkable target defined before the start. Constraints in a file rather than in chat. A commit at every improvement, with the check result in the message. A budget with a defined exit that produces a written report. A CI check that refuses a weakened test suite. And a human who glances at the checkpoint log twice and otherwise does something else.

Set up that way, the worst outcome is a bounded amount of compute and a report about what does not work, which is a normal engineering result. Set up without it, the worst outcome is a large confident diff that passes its own tests and does not solve the problem, which is considerably worse than nothing, because it takes someone a day to work out what happened.

Give the run a way to ask

An unattended run is not obliged to be entirely unattended. A run that can leave a question — write it to a file, post it to a channel, and carry on with a different part of the task — behaves considerably better than one that must either guess or halt.

The pattern is: on encountering a decision it should not make alone, record the question and the assumption it is proceeding under, then continue. You get a list at the end of every place the agent guessed, ranked by how consequential it thought the guess was, which is a far better review input than a diff alone.

The cost profile is unlike anything else in your toolchain

A two-hour agent run is not a two-hour build. It is a sequence of model calls each carrying an increasingly long context, so cost per step rises as the run continues, and a struggling run costs more per unit of progress precisely because it is struggling.

That is worth knowing when setting the budget, because the intuition from CI — longer job, linearly more cost — understates it. The last twenty minutes of a long run are the most expensive twenty minutes in it, and they are also the least likely to be productive, which is a good argument for budgets that are tighter than they feel.

Reviewing the output of a long run

One practical instruction. Do not review the final diff against the original request; review it against the checkpoints. The diff from the last good checkpoint to the end is where the drift lives, and it is usually a small fraction of the total change while containing most of the risk.

That reframing turns an intimidating two-hundred-file review into a targeted one: the early checkpoints were verified by the check passing at the time, and the interesting question is what happened after progress stopped.

The organisational failure mode

One thing to watch for beyond the technical. Long autonomous runs produce large diffs, and large diffs produce shallow reviews. A team that adopts multi-hour runs without changing anything about review will approve more code with less scrutiny, and the effect is invisible because every individual review felt normal.

The counterweight is to insist that a long run's output is decomposed before review — split into the checkpoints, or into separate pull requests by concern — rather than reviewed as one artefact. That costs the agent a few minutes at the end of a run and saves a reviewer from a document they cannot honestly assess.

What justifies the setup cost

Everything in this article is overhead: a target check, a checkpoint discipline, a constraints file, a budget with an exit, a test-integrity gate. That is perhaps a day of work to establish, and it is not worth doing for a team that runs one long session a month.

It becomes worth it the moment long runs are routine, because the failure this article describes is not occasional. Productive-looking motion is the default behaviour of a loop with no futility signal, so the question is not whether it will happen but how much of it you will pay for before you notice. A day of setup against that is a good trade, and it is a much easier trade to make before the first two-hour run that produced nothing than after.