Measuring AI Impact: Ask the Questions That Have Answers

There is no control group. You cannot run the same quarter twice. Organisations that try to answer the big question anyway produce a number, defend it, and learn nothing.

The question that cannot be answered, and the ones that can

"Are agents making us faster?" is not empirically tractable at a single company. There is no control group — you cannot run the same quarter twice, and the team, the codebase and the market all changed alongside the tooling.

Organisations that try to answer it anyway produce a number, defend it, and learn nothing. The productive move is to abandon the totalising question and ask narrower ones that comparison can actually settle.

Every question on the right compares things that coexist, which is the only comparison this data supports.

Use the delivery metrics you already have

Four widely-used delivery measures cover most of what matters, and they predate agents entirely — which is their advantage. They were not designed to make a tool look good.

Lead time is the one that most often surprises people. Authoring gets faster and review gets slower, and the second frequently dominates — which shows up as lead time flat or worse despite everyone feeling more productive.

Instrument once, at the pull request

## Agent involvement
- [ ] None
- [ ] Assisted — agent wrote part, I directed and revised
- [ ] Generated — agent produced it, I reviewed

Prompt (if unattended):
Tokens (if known):

Three checkboxes, and every question above becomes a query against data you already collect. Define the middle category precisely or it absorbs everything — "assisted" should mean the agent wrote code that shipped, not that someone asked it a question.

SELECT agent_involvement,
       COUNT(*)                                  AS prs,
       ROUND(AVG(hours_open), 1)                 AS avg_lead_hours,
       ROUND(AVG(review_rounds), 2)              AS avg_rounds,
       ROUND(AVG(CASE WHEN reverted_within_30d
                      THEN 1.0 ELSE 0 END), 3)   AS failure_rate
FROM pull_requests
WHERE merged_at > now() - interval '90 days'
GROUP BY agent_involvement;

Three ways this goes wrong

Selection effects

People give agents the tasks they believe agents are good at. Comparing agent-assisted to unassisted work compares two different populations of task, not two methods. A lower failure rate for agent-assisted PRs may only mean agents get the dependency bumps and humans get the hard problems.

You cannot eliminate this. You can control for it crudely by comparing within a task category — agent-assisted dependency upgrades against unassisted dependency upgrades — which is narrower and much more honest.

Measuring individuals

The moment these numbers attach to people they become targets and stop measuring anything. Aggregate at team level, say so explicitly before anyone asks, and mean it. A single instance of individual measurement permanently poisons the data, because people optimise the metric rather than the work.

Goodhart on volume

Anything that counts output — PRs merged, lines changed, commits — rewards splitting work and discourages deletion. The best change is often a deletion, and it scores negatively on every volume measure.

Cost, since somebody will ask

Token spend is the easiest number to produce and the least informative on its own. The denominator decides whether it means anything.

The last row is the one that changes decisions. Learning that dependency upgrades cost a dollar of tokens per merged PR and multi-file feature work costs forty tells you where delegation is worth it, which is a more useful answer than any aggregate.

Keep the qualitative half

Numbers tell you something changed. They do not tell you why, and the why is where the action is.

Where did an agent save you real time this fortnight, and where did it waste your time?

The second half is the valuable one and the half people volunteer least readily — admitting you spent an hour untangling agent output feels like admitting you used the tool badly. Asking it as a routine question with no judgement attached surfaces patterns no metric will: a task category that consistently disappoints, a repository where output is worse, a context-file rule that is actively misleading.

Reporting it upward without overclaiming

Leadership will ask for a number. The honest answer has a shape worth rehearsing, because the alternative is inventing a productivity percentage that nobody can defend and everyone remembers.

Agent-assisted changes are 31% of merged PRs this quarter. Within dependency upgrades, they take a similar number of review rounds and have the same 30-day revert rate as unassisted ones, at roughly a dollar of tokens each. Within multi-file feature work, they take about one extra review round. We have not measured an overall speed change, and we do not think that question is answerable here.

That last sentence is the one to keep. Saying plainly that a question is not tractable is more credible than a confident number, and it protects the findings that are solid from being dismissed alongside a figure somebody successfully attacks.

The smallest version worth doing

1. Add the three checkboxes to your PR template.
2. Wait ninety days. Change nothing else.
3. Compare review rounds and 30-day revert rate across the three categories.
4. Ask the fortnightly question in retrospectives and write down the answers.

That is nearly free, and after a quarter you have evidence where most organisations have conviction. Which is a lower bar than it sounds, and a genuinely better position to make the next decision from.