Running Coding Agents in CI: Guardrails for Unattended Work

CI runners hold your deploy keys and registry tokens, because that is what CI is for. An agent running there inherits the most powerful credentials you have, unsupervised.

What changes when nobody is watching

An agent in your terminal has a human in the loop by default. You see each step, and you interrupt when it goes wrong. In CI nobody is watching, and three things that were manageable become structural problems.

• Errors compound. A wrong turn at step three is caught immediately by a human and not at all by a pipeline.
• Cost is unbounded. An agent that retries indefinitely does so on your budget, at machine speed.
• Credentials are richer. CI runners hold deploy keys, registry tokens and production access, because that is what CI is for.

The last is the one to design around first. It is not a small increase in exposure — it is the most powerful credential set in the organisation, in an environment with no live supervision.

Tasks that work unattended

The common property is that success is machine-checkable.

Tasks that do not work unattended share the opposite property: success is a judgement. "Improve the error handling" has no green check to converge on, so an unattended agent optimises for the appearance of completion.

The pattern that holds

name: Agent dependency upgrade
on:
  schedule: [{cron: '0 3 * * 1'}]
  workflow_dispatch:

permissions:
  contents: write        # branch only — main is protected
  pull-requests: write
  # deliberately absent: packages, deployments, id-token

jobs:
  upgrade:
    runs-on: agent-runner        # dedicated, not the deploy runner
    timeout-minutes: 30          # hard ceiling
    steps:
      - uses: actions/checkout@v4
      - name: Run agent
        env:
          AGENT_MAX_TOKENS: 400000
        run: ./ci/agent-task.sh "Upgrade deps in package.json to latest
                                 minor. Fix any resulting test failures.
                                 Do not change application behaviour."
      - name: Verify
        run: ./check.sh          # exactly what a human PR must pass
      - name: Open PR
        run: gh pr create --draft --label agent-generated

Five controls in that file, each closing a specific failure:

• Narrow permissions — no publish, no deploy, no OIDC token. The most powerful CI credentials are simply not present.
• Dedicated runner — not the one your deploy pipeline uses, so the agent never shares an environment with production access.
• Timeout — converts "stuck forever" into "failed after thirty minutes".
• Token ceiling — bounds cost independently of time.
• Draft PR, labelled — output enters review as something a human must act on, clearly marked.

Prompts for unattended runs

Interactive prompts can be vague, because you correct course as you go. CI prompts cannot.

Two properties make the right column work: an explicit scope boundary, and an explicit completion criterion. Without a boundary an unattended agent expands the task; without a criterion it does not know when to stop.

Failure needs to be legible

An unattended run that fails should say why in a form a human can act on without replaying the session. The default — a red X and a thousand lines of log — means nobody investigates, and the job quietly becomes noise people mute.

- name: Summarise outcome
  if: always()
  run: |
    {
      echo "## Agent run"
      echo "- exit: ${{ job.status }}"
      echo "- tokens: $(cat .agent/tokens 2>/dev/null || echo n/a)"
      echo "- duration: ${SECONDS}s"
      echo "- files changed: $(git diff --name-only | wc -l)"
      echo ""
      echo "### Last check output"
      tail -30 .agent/check.log 2>/dev/null || echo "no output"
    } >> "$GITHUB_STEP_SUMMARY"

Four categories of failure are worth distinguishing in that summary, because each calls for a different response: hit the timeout, hit the token ceiling, finished but the checks failed, or errored outright. Only the third is about the code. The other three are about the job configuration, and treating them all as "the agent failed" sends people looking in the wrong place.

Marking the output honestly

Agent-generated PRs should be identifiable. Not because they are inherently worse, but because reviewers should calibrate differently — the author cannot answer questions, and no human formed an intent behind any particular line.

• Label them agent-generated.
• Open as draft, so nothing merges by momentum.
• Put the exact prompt in the PR description. Reviewers need to know what was asked to judge whether it was answered.
• Require an approving review from a human who is not the person who triggered the job.

Cost control that actually binds

An unattended agent has no natural stopping point. Three limits, and they catch different failures:

Set all three. They are cheap to configure and each covers a gap the others leave. A job that hits a limit should fail loudly rather than partially commit — half-finished work merged because the timeout fired mid-task is worse than no work at all, and considerably harder to diagnose.

Start with the boring one

If you are adding agents to CI for the first time, start with weekly dependency upgrades. Well-defined, machine-verifiable, low stakes, genuinely useful, and a task nobody enjoys.

It also exercises the whole path — permissions, runner isolation, timeouts, PR creation, review flow — on something where getting it wrong costs you a closed pull request. Learn the mechanics there before pointing the same machinery at anything that matters.