Instrumenting Agent Sessions: Metrics That Aren't Vanity

Ask a team whether agents are helping and you get confident answers in both directions from people in the same repository. Both are sincere. Neither is evidence.

Opinions where measurements should be

Ask a team whether coding agents are helping and you will get confident answers in both directions from people working in the same repository. Both are sincere; neither is evidence. Nobody is measuring anything.

This matters because the interesting decisions — which tasks to hand over, how much context to write, whether a tool change helped — are empirical, and answering them by argument produces whatever the most confident person believes.

Why teams avoid this

Worth naming the reason measurement here is rare, because it is not laziness.

Measuring might produce an unwelcome answer. A team that has invested in agent workflows, told its leadership about the gains and built process around them has an uncomfortable amount riding on the numbers coming out well. The safest thing is to not look.

The counter is that the decisions you actually face are narrow ones — which task types to hand over, whether last month's context file rewrite helped — and those have unthreatening answers. "Agents work well for dependency upgrades and poorly for schema migrations" is useful and costs nobody anything. It is only the totalising question, are agents worth it, that carries political risk, and that is the question the data cannot answer anyway.

Metrics that mislead

Start with what not to count, because these are the ones that get proposed first.

They share a defect: each measures agent activity rather than outcome. Activity is easy to count and easy to increase without improving anything.

The comparison problem

Before any of this, one methodological point that undermines most attempts at measurement here.

There is no control group. You cannot run the same quarter twice, once with agents and once without. Teams change, codebases change, people learn. Any before-and-after comparison spanning months is comparing two different organisations and attributing the difference to one variable.

This does not make measurement pointless. It changes what the measurement is for. You are not answering "are agents worth it" — that question is not empirically tractable at a single company. You are answering narrower questions that are:

• Did rework increase after we started using agents for this task type?
• Did completion rate improve after we rewrote the context file?
• Are agent-assisted PRs taking longer to review than unassisted ones, right now?

Each compares things that coexist rather than things separated by months, which is the only comparison this data can honestly support.

What is worth counting

Rework rate

Of changes that reach a pull request, what fraction need substantial rework before merging? Track it separately for agent-assisted and unassisted changes. This is the single most informative number available, because it captures quality at the point where quality becomes visible.

Review burden

Time from PR opened to approved, and number of review rounds. If agents are producing more code that takes longer to review, throughput has not improved — it has moved.

Defect rate after merge

Bugs traced to changes shipped in the last quarter, split by whether an agent was involved. Lagging, noisy, and the one that actually answers the question people are arguing about.

Task completion without intervention

For unattended runs, what fraction reach a passing state alone? This is the metric to watch when tuning context files, because it responds to changes quickly enough to be useful.

Cost per merged change

Tokens spent divided by changes actually merged. Note the denominator: cost per session rewards abandoning sessions early, and cost per merged change does not.

Instrumenting it

Most of this comes from data you already have, joined on one new field.

## Agent involvement
- [ ] None
- [ ] Assisted — agent wrote part, I directed and revised
- [ ] Generated — agent produced it, I reviewed

Tokens (if known):
Prompt (if unattended):

Three checkboxes, filled honestly, and every metric above becomes a query against your existing PR data. The middle category is the one that needs defining clearly, or it absorbs everything.

-- Rework rate by involvement, last 90 days
SELECT agent_involvement,
       COUNT(*) AS prs,
       AVG(review_rounds) AS avg_rounds,
       AVG(CASE WHEN commits_after_first_review > 3
                THEN 1.0 ELSE 0.0 END) AS heavy_rework_rate
FROM pull_requests
WHERE merged_at > now() - interval '90 days'
GROUP BY agent_involvement;

Interpreting it honestly

Three cautions, because this data is easy to misuse.

Selection effects dominate. People hand agents the tasks they think agents are good at. Comparing agent-assisted to unassisted work compares different populations of task, not different methods.

Never measure individuals. The moment these numbers are attached to people, they become targets and the data stops meaning anything. Aggregate at the team level and say so out loud, before anyone asks.

Direction beats level. Whether your rework rate is 18% or 24% matters far less than whether it moved after you changed something. Watch the trend across a change you made deliberately.

Qualitative signal is not worthless

Numbers answer "did it change". They do not answer "why", and the why is where the actionable material usually is.

A short recurring question in retrospectives does more than it looks like it should:

Where did an agent save you real time this fortnight, and where did it waste your time?

The second half is the valuable one, and it is the half people volunteer least readily — admitting you spent an hour untangling agent output feels like admitting you used the tool badly. Asking directly, as a normal question with no judgement attached, surfaces patterns that no metric will: a category of task that consistently disappoints, a repository where output is worse, a context file rule that is actively misleading.

Pair it with the quantitative data rather than choosing between them. The numbers tell you something changed; the conversation tells you what to do about it.

The smallest useful version

If a full programme is more than you want, do this:

1. Add the three checkboxes to your PR template.
2. Wait ninety days.
3. Compare review rounds and post-merge defects across the three categories.

Nearly free, and after a quarter you have evidence instead of opinions. That alone puts you ahead of most teams currently having the same argument with more confidence and less data.