CI Gating for Agent-Generated Pull Requests

Agents are good at satisfying mechanical checks, because mechanical checks are the feedback they iterate against. Your gate confirms less than it used to.

The gate is doing something different now

CI has always answered one question: does this change break anything? That question is necessary and it was calibrated for a world where a human wrote every line and understood why.

Agent-generated pull requests satisfy the old gate more easily than human ones. The code compiles, the tests pass, the linter is clean — agents are good at satisfying mechanical checks, because mechanical checks are exactly the feedback they iterate against. What the gate no longer confirms is that anybody understood the change.

So the gate needs to answer a second question: is this change reviewable, and has it been reviewed by someone who could have caught what the machine cannot?

What to add

None of these are agent-specific in principle. All of them matter more when change volume rises and the author cannot be questioned.

A workable gate

name: PR gate
on: pull_request

jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }

      - name: Standard checks
        run: ./check.sh

      - name: Diff size
        run: |
          N=$(git diff --shortstat origin/${{ github.base_ref }}...HEAD \
              | grep -oE '[0-9]+ insertion' | grep -oE '[0-9]+' || echo 0)
          echo "Added lines: $N"
          if [ "$N" -gt 600 ] && \
             ! ${{ contains(github.event.pull_request.labels.*.name,
                            'large-change-approved') }}; then
            echo "Over 600 added lines. Split it, or add the"
            echo "'large-change-approved' label with a reason."
            exit 1
          fi

      - name: Tests changed with source
        run: |
          CHANGED=$(git diff --name-only origin/${{ github.base_ref }}...HEAD)
          if echo "$CHANGED" | grep -q '^tests/' && \
             echo "$CHANGED" | grep -q '^src/'; then
            echo "::warning::Source and tests changed together."
            echo "Confirm the tests were not adjusted to fit the code."
          fi

      - name: No new suppressions
        run: ./ci/no-new-suppressions.sh

      - name: Protected paths
        run: ./ci/protected-paths.sh

Two design choices in there are worth naming. The diff-size limit has a labelled override rather than a hard stop, because some changes legitimately are large and a rule with no escape gets deleted. And the test-alongside-source check emits a warning rather than failing, because that combination is often perfectly correct — the point is to make a reviewer look, not to block.

Coverage on changed lines, not overall

Total coverage percentage is a poor gate — it moves slowly, and a large change with no tests barely shifts it. Coverage on the lines this change touched is the useful measure and most tooling supports it.

- name: Coverage on changed lines
  run: |
    pytest --cov=src --cov-report=xml -q
    diff-cover coverage.xml \
      --compare-branch=origin/${{ github.base_ref }} \
      --fail-under=80

This catches the specific pattern where an agent adds a well-tested feature and one untested error path — invisible in total coverage, obvious in diff coverage, and the untested path is usually the one that matters.

Keeping the gate fast

A gate that takes twenty minutes gets worked around. People push, switch tasks, forget, and come back to a red build with no memory of the change. With higher change volume, that cost multiplies.

Split it into two stages so feedback arrives while the change is still in mind:

Fast stage — under 90 seconds, runs on every push
  lint, format check, type check, unit tests,
  diff size, suppressions, protected paths

Slow stage — runs on PR open and on demand
  integration tests, coverage, build,
  security scan, dependency audit

The cheap structural checks belong in the fast stage deliberately. A diff size or suppression failure should arrive in ninety seconds, not after the integration suite, because those are the ones where the fix is to restructure the change rather than to debug it.

Require a human, and say why

The most important rule in the gate is not a script. Branch protection requiring an approving review from someone other than the PR author is what keeps a fully automated path from existing at all.

For agent-generated changes specifically, two additions are worth making:

• The approver must not be whoever triggered the agent. Otherwise one person can run an agent and approve its output, which is the automated path with extra steps.
• The PR description must contain the prompt. A reviewer cannot judge whether a change answers the question without knowing the question. Enforce it with a template check if you have to.

The label discipline

Several checks above use labels as escape hatches, which only works if labels mean something. Two rules keep them honest.

Labels require a written reason. A bare large-change-approved label is a rubber stamp. The same label with a comment explaining why the change cannot be split is a decision someone made and signed.

Count them. If override labels are applied to a third of your pull requests, the threshold is wrong, not the changes. Review the counts monthly and adjust the limit rather than watching people route around it.

gh pr list --state merged --limit 200   --json number,labels,mergedAt   --jq '.[] | select(.labels[]?.name
        | test("approved|reviewed|override"))
        | [.mergedAt, .number, (.labels|map(.name)|join(","))]
        | @tsv'

Do not gate on model-graded review

A tempting addition is a second model reviewing the first model's output as a required check. Useful as an advisory comment; poor as a gate.

It shares failure modes with the model that wrote the code, so it misses correlated errors — the very ones you most need caught. It produces confident-sounding comments on trivia, which trains reviewers to skim. And worst, it creates the impression that review happened, which is the specific belief that lets unreviewed code merge.

Run it if you like. Post it as a comment. Do not let it satisfy the approval requirement, and do not let anyone describe it as "reviewed".