Benchmarking Autocomplete: Measure Latency, Not Completion Quality

A suggestion arriving after you typed the next line is worthless however good it was. Any benchmark not measuring time-to-first-token on your hardware is measuring something else.

Benchmarks that answer the wrong question

Comparisons of local and hosted models for autocomplete usually report completion quality on a benchmark suite. That number is close to irrelevant for this workload, because inline completion is not a capability problem.

It is a latency problem. A suggestion that arrives after you have typed the next line is worthless regardless of how good it was, and one that arrives in 80 ms is useful even when mediocre. Any benchmark that does not measure end-to-end time-to-first-token on your own hardware is measuring something else.

What to actually measure

The last row is the one to build if you build only one. Acceptance rate measures whether a suggestion looked right at the moment of pressing Tab. Retention — whether the accepted text is still there thirty seconds later — measures whether it actually was. The two diverge more than people expect, and optimising acceptance alone optimises for plausibility.

Measure the whole path

The common error is benchmarking inference in isolation. The user experiences everything between keystroke and rendered suggestion, and inference is one term in that sum.

keystroke
  → debounce                    typically 50–150 ms, and yours to tune
  → context assembly            reading nearby files, building the prefix
  → serialise + network         0 ms local; 20–120 ms hosted, by region
  → queue wait                  the term that dominates under load
  → prefill                     scales with prompt length
  → first token
  → remaining tokens
  → render

Two of those terms are worth attention before anyone compares models. Debounce is pure configuration and frequently mistuned — a 200 ms debounce makes a 60 ms model feel like a 260 ms one. And prefill scales with how much context you send, so a client that ships 8k tokens of surrounding code pays for that on every keystroke.

A harness worth running

import json, statistics, time
import httpx

# Real prefixes from your own codebase, not synthetic snippets.
CASES = json.load(open("bench/prefixes.json"))

def measure(endpoint: str, payload: dict) -> float:
    start = time.perf_counter()
    with httpx.stream("POST", endpoint, json=payload, timeout=10) as r:
        for chunk in r.iter_bytes():
            if chunk.strip():
                return (time.perf_counter() - start) * 1000   # ms to first token
    return float("inf")

for name, endpoint in [("local", LOCAL), ("hosted", HOSTED)]:
    samples = []
    for case in CASES:
        for _ in range(20):            # variance is large; one run tells you nothing
            samples.append(measure(endpoint, {"prefix": case["prefix"],
                                              "suffix": case["suffix"],
                                              "max_tokens": 64}))
            time.sleep(0.05)
    samples.sort()
    print(f"{name:8s} p50={statistics.median(samples):6.1f}ms "
          f"p95={samples[int(len(samples)*0.95)]:6.1f}ms "
          f"n={len(samples)}")

Conditions that change the answer

A single number hides the cases that determine whether people keep the feature switched on.

• Under load. A hosted endpoint at p50 on a quiet Tuesday is not the same endpoint at 10am when your whole team is typing. Measure concurrently.
• On a laptop doing other work. A local model sharing a GPU with a running test suite or a video call behaves differently from one on an idle machine.
• On the worst network anyone has. Someone is on hotel wifi. Hosted p95 for them is the number that generates the complaint.
• Cold. First request after idle. Local models may need to page weights back in; hosted endpoints may have scaled down.

Reading the result honestly

The likely outcome, and worth predicting so you notice if yours differs: local wins decisively on latency, hosted wins on multi-line quality, and the crossover is around how much of the suggestion you want.

Which points at a hybrid rather than a choice: local for the inline path, hosted for anything invoked deliberately. That is not a compromise; it matches the two workloads to the two constraint profiles.

Reporting it so it changes a decision

A benchmark that produces one number per model gets argued about. One that produces a distribution per condition gets acted on.

                        p50      p95      p99    accept   retain
local-7b   idle         62ms    118ms    190ms     31%      24%
local-7b   under load  104ms    240ms    410ms     31%      24%
hosted     idle        180ms    340ms    620ms     38%      33%
hosted     under load  210ms    520ms   1400ms     38%      33%
hosted     poor wifi   340ms    980ms   3100ms     38%      33%

n = 4,000 completions per row, real cursor positions, 3 days

Read the p99 column. That is where people form opinions — the occasional three-second wait is what someone remembers and complains about, not the median they never notice. A model that wins on p50 and loses badly on p99 will be experienced as the worse one.

The context you send is a variable too

One factor sits between the two models and is often the largest lever: how much surrounding code the client ships with each request. Prefill scales with it, so a client sending 8k tokens of context pays that on every keystroke.

prefix tokens    local p50    hosted p50
   512             48 ms       165 ms
 2,048             71 ms       190 ms
 8,192            186 ms       255 ms

Two things fall out of a table like that. The local model degrades faster with context, because prefill is compute-bound and your GPU is smaller. And there is usually a knee — a context size beyond which quality stops improving while latency keeps rising. Find yours before comparing models, because otherwise you are comparing two different context configurations.

The cost model is different too

Autocomplete generates enormous request volume — potentially one per keystroke pause, per developer, all day. Hosted inference is priced per token, so the marginal cost is real and scales with team size. Local inference is capital: hardware once, then free.

That inversion matters more here than for any other agent workload. A per-token cost that is trivial for a handful of deliberate agent sessions becomes significant when the same model is invoked thousands of times an hour per person, which is exactly what inline completion does.