Screenshot-Driven UI Debugging: The Picture Is Evidence, Not a Diagnosis

The agent adds overflow: hidden and the screenshot looks right. The bug moved; it did not go away.

A screenshot tells the agent what, never why

Dragging a picture of a broken layout into an agent session works, and it works well enough that it is easy to over-trust. The model can see that a card overflows its container, that two columns are misaligned by a few pixels, that a label is clipped, that text sits on a background it cannot be read against.

What it cannot see is the cascade. The pixels are the output of a computation over hundreds of declarations, inheritance, specificity, and box model decisions made three ancestors up. An agent looking only at the rendering is doing what you do when you squint at a page and guess — which is a fine way to form a hypothesis and a poor way to make a change.

The failure this produces is specific and recognisable: the agent adds overflow: hidden, or a negative margin, or !important, and the screenshot now looks right. The bug moved. It did not go away.

The bugs vision genuinely catches

Purely visual defects, where the rendering is the whole truth. Overflow and clipping. Alignment across a row. Z-order and unexpected stacking. Contrast that fails against a background. Spacing that is inconsistent between otherwise identical cards. A font that did not load and fell back. Something rendering at the wrong size on a narrow viewport.

For these, an image is the most efficient possible input: describing them in prose takes three paragraphs and loses information the picture carries exactly.

The bugs it does not

Anything whose cause is a rule. A parent with overflow: hidden that clips a dropdown. A specificity collision where a utility class silently loses. A flex child that will not shrink because its default min-width: auto is holding it open. A stacking context created by a transform two levels up, which is why your z-index: 9999 did nothing.

Every one of those looks, in a screenshot, like a plain misalignment. The image contains no evidence for the actual cause, so an agent working from the image alone is guessing about mechanism from appearance.

Send the computed styles with the picture

This is the whole technique. Capture the rendering and the resolved state of the relevant subtree at the same moment, and hand the agent both. Now the picture says what is wrong and the dump says what the browser actually decided.

import { chromium } from "playwright";

const [url, selector] = process.argv.slice(2);
const browser = await chromium.launch();
const page = await browser.newPage({ viewport: { width: 1280, height: 900 } });

await page.goto(url, { waitUntil: "networkidle" });
await page.emulateMedia({ reducedMotion: "reduce" });
await page.evaluate(() => document.fonts.ready);

await page.locator(selector).screenshot({ path: "bug.png" });

const styles = await page.evaluate((sel) => {
  const props = [
    "display", "position", "overflow", "z-index", "transform",
    "flex", "min-width", "width", "height", "margin", "padding",
    "box-sizing", "grid-template-columns", "align-items",
  ];
  const out = [];
  let el = document.querySelector(sel);
  while (el && el !== document.documentElement) {
    const cs = getComputedStyle(el);
    const row = { el: el.tagName.toLowerCase() + "." + [...el.classList].join(".") };
    for (const p of props) row[p] = cs.getPropertyValue(p);
    const r = el.getBoundingClientRect();
    row.rect = [r.x, r.y, r.width, r.height].map(Math.round);
    out.push(row);
    el = el.parentElement;
  }
  return out;
}, selector);

console.log(JSON.stringify(styles, null, 1));
await browser.close();

The ancestor walk is the part that matters. Most layout bugs that survive a first fix attempt are caused by an ancestor, and the ancestor chain is exactly what a screenshot of the broken element omits.

Determinism, or you are debugging the capture

An unstable screenshot wastes turns on differences that are not the bug. Pin the viewport. Wait for fonts — document.fonts.ready, not a sleep. Disable animations and transitions. Freeze anything time-dependent: relative timestamps, randomised placeholder content, carousels. Seed your fixtures.

If you skip this, the agent will confidently explain a two-pixel shift that is really a font swapping in late, and you will both spend a turn on it.

Close the loop with the after image

The half of this workflow people skip is capturing again after the change. An agent that proposes a fix and never sees the result is working open-loop, and open-loop is where confident wrong answers come from. The same script, run again, gives it the evidence to judge its own work — which is the general pattern argued for in TDD-style agentic loops, with a screenshot standing in for the assertion.

Once you have before-and-after captures, the natural next step is to keep them: a visual regression baseline, so the fix that stops this bug also stops its recurrence. That turns a debugging session into a test, which is the only way the effort compounds.

Where this leaves the screenshot

Useful, and not sufficient. It is the fastest way to communicate what is wrong and the slowest way to establish why. Used as the sole input it produces plausible patches that move bugs around. Paired with resolved styles and an after-capture it turns into an actual diagnostic loop.

Responsive bugs need more than one capture

A single screenshot describes one viewport. Most layout bugs that reach production are not present at the width the developer works at — they appear at 390 pixels, or at the awkward band just before a breakpoint fires, or at 1440 with a scrollbar taking 15 pixels the design did not account for.

Capture a strip rather than a frame: three or four widths that bracket your breakpoints, in one run, handed over together. An agent given the set can see which side of a breakpoint the failure starts on, which is most of the diagnosis. An agent given the broken width alone will change a value that fixes it there and breaks the width you did not send.

import { chromium } from "playwright";

const url = process.argv[2];
const widths = [390, 744, 1024, 1280, 1440];

const browser = await chromium.launch();
for (const w of widths) {
  const page = await browser.newPage({ viewport: { width: w, height: 900 } });
  await page.goto(url, { waitUntil: "networkidle" });
  await page.emulateMedia({ reducedMotion: "reduce" });
  await page.evaluate(() => document.fonts.ready);
  await page.screenshot({ path: `shot-${w}.png`, fullPage: true });
  await page.close();
}
await browser.close();

The states a screenshot cannot reach

Hover, focus, active, disabled, loading, error, and every state behind a condition your fixture does not trigger. These are where visual bugs concentrate, precisely because they are hard to capture and therefore rarely reviewed.

Playwright can force most of them — page.hover() before the capture, locator.focus(), a route interception that returns a 500 so you can see the error state, an artificially delayed response for the loading state. The work is in deciding which states matter, not in producing them. A component with six states and one screenshot has been reviewed at about seventeen percent.

When the picture is not of a web page

The technique degrades gracefully but it does degrade. A screenshot of a native application, a PDF, or a design tool gives the agent the pixels and nothing else — there is no computed-style equivalent to pair with it, so you are back to reasoning from appearance alone.

For those, invert the workflow: instead of asking what is wrong with the rendering, ask the agent to describe what it sees, and check the description against what you intended. A mismatch in the description locates the problem faster than a proposed fix does, because it tells you whether the agent is even looking at the thing you are asking about. It is a slower loop, and it is honest about its own uncertainty, which the confident-patch loop is not.

Turning the debugging session into a regression test

The part of this workflow that compounds is the last step, and it is the one that gets skipped because the bug is already fixed and everyone wants to move on.

You have, at the end of a successful session, a capture script pinned to a viewport with fonts and animation controlled, a selector that isolates the component, and a correct rendering. That is a visual regression test with about four lines of glue. Add it and this specific bug cannot come back silently; skip it and you will debug a variant of it in four months, from scratch, with a fresh screenshot.

import { test, expect } from "@playwright/test";

test.use({ viewport: { width: 1280, height: 900 } });

test("summary card does not overflow at long titles", async ({ page }) => {
  await page.goto("/fixtures/card?title=long");
  await page.emulateMedia({ reducedMotion: "reduce" });
  await page.evaluate(() => document.fonts.ready);

  await expect(page.locator("[data-test=summary-card]"))
    .toHaveScreenshot("summary-card-long-title.png", { maxDiffPixels: 24 });
});

The fixture route is the piece worth arguing for. Pointing a visual test at a real application page makes it fail whenever anything on that page changes, which trains everyone to update baselines without looking. Pointing it at a route that renders one component with fixed props makes it fail only when that component changes, which is what you wanted. A maxDiffPixels tolerance absorbs antialiasing differences without absorbing a real shift.

Baselines are only useful if regenerating them is deliberate

Every visual regression suite dies the same way: a legitimate design change makes forty snapshots fail, someone regenerates all of them, and one real regression is committed alongside the intended change. After that the suite is noise, and people stop reading it.

The defence is procedural rather than technical. Regenerating baselines is its own commit, containing nothing else, with the rendered before-and-after attached. It is reviewable in the way an ordinary code change is reviewable, because the diff is a set of images someone has to look at and approve. That is slower, and it is the only version that keeps working past month three.

An agent can run all of this — capture, compare, propose, re-capture — without supervision. What it should not do unsupervised is approve a new baseline, for exactly the reason above: accepting a changed rendering is a judgement about intent, and intent is the one input the agent does not have.

What the picture is actually for

Reduced to one sentence: the screenshot is how you tell the agent what you mean, and the computed styles are how it finds out what is true.

Most of the frustration people report with vision-assisted UI work comes from collapsing those two roles. A picture is an excellent specification — "this, but the card should not overflow" carries more meaning in one image than in a paragraph. It is a poor observation, because the causal information is not in it. Asking an image to serve as both produces an agent that is confident about a mechanism it inferred from appearance, and the patches follow from there.

Keep the roles separate and the workflow becomes ordinary: you specify with the picture, the agent observes with the tooling, proposes a mechanism, makes a change, and re-observes. That is the same loop as any other debugging, with one unusual input format, and it behaves like any other debugging once the inputs are honest about what they contain.