Local and Hosted Models: Deciding on Data Flow, Not Benchmarks

Most local-versus-hosted debates are settled by reading the terms you are already on. It is a ten-minute check that frequently dissolves the entire argument.

Reading order matters

One habit worth adopting: read the tests before the implementation.

With a human author you often read the implementation first, because you trust that their tests reflect their intent. With an agent the tests are where intent was most likely lost — a test asserting current behaviour rather than correct behaviour looks identical to a good test until you compare it against what was asked.

Reading tests first also gives you the specification the agent worked to. If the tests describe something narrower than the request, the implementation is probably narrower too, and you have found the gap before spending attention on code that was answering the wrong question.

Comparisons of local versus hosted models usually turn on benchmark scores, which is the least useful axis for this decision. The useful question is what each arrangement does to your context — what leaves your machine, what can be retained, and what you are able to prove about it.

What actually leaves

Worth being precise, because people are frequently wrong in both directions.

The distinction between rows two and three matters more than the local versus hosted split for most organisations. A business agreement with a defined retention policy and no training use satisfies the majority of real requirements, and the difference from the consumer tier is a contract rather than a technology.

What leaves is not only the prompt

People assess this by thinking about the code in the prompt. Several other things travel with it, and they are easy to overlook.

• File paths and structure. Directory layout, module names and internal service names reveal architecture even when no file content is sent.
• Tool results. A database schema dump, an error message containing a hostname, a config listing. These leave with the same request.
• Your context file. AGENTS.md is sent every turn, and it frequently names internal systems, security constraints and deployment details.
• Test fixtures. If fixtures contain realistic customer data — which is common and rarely audited — that data leaves whenever a test file is read.

The last is the one that catches organisations out, because fixtures are rarely reviewed with the same care as production code paths and quite often began life as a copy of real data.

Where local genuinely wins

• A hard prohibition on egress. Some regulated environments forbid it outright. This is the one case where the decision is made for you.
• Air-gapped work. No connectivity, no choice.
• Very high volume of low-value calls. Local inference has no marginal cost, which changes what is worth doing at all.
• Latency-critical inline completion. A local small model beats a network round trip for single-line suggestions.

Where hosted still wins

• Capability on hard tasks. Frontier hosted models remain ahead of what runs on a laptop, and the gap is largest on exactly the multi-step reasoning that agentic work requires.
• Context length. Large windows are memory-hungry; local models are usually far more constrained, which bites on repository-scale work.
• No operational burden. Nobody maintains inference infrastructure, chases driver issues or plans capacity.

Cost behaves differently

The two arrangements have opposite cost curves, and it changes what is sensible to do with each.

Hosted inference is marginal: every call costs, so high-frequency low-value uses are hard to justify. Local inference is capital: hardware is bought once, and the marginal call is free.

That inversion makes things viable locally that would never survive a per-token budget — generating a commit message on every commit, drafting a summary of every diff, classifying every incoming issue. None of those are worth a hosted call individually. All of them are worth doing when the call is free.

The hybrid that works in practice

Splitting by task rather than choosing globally is the arrangement most teams end up at, and it is not a compromise — it is a better fit for the actual distribution of work.

Row four is the important one, and it needs a mechanism rather than a convention. "Remember to switch models when touching sensitive code" is not a control; it is a hope. If some data must not leave, the tooling has to enforce that, ideally by making the hosted path unavailable in those repositories.

Latency is a capability, not a comfort

One factor that gets treated as convenience and is not: for inline completion, latency determines whether the feature is usable at all.

A suggestion arriving in 80 ms is part of typing. The same suggestion at 600 ms arrives after you have moved on, and you spend more time reading and dismissing it than it saves. Past roughly a quarter of a second, quality stops compensating — a better suggestion that arrives too late is worse than a mediocre one that arrives in time.

This is the strongest argument for local models, and it has nothing to do with privacy. A small local model with no network round trip beats a frontier hosted model on the specific task of completing the line you are currently typing, because the task is latency-bound rather than capability-bound.

The inverse holds for agentic work. A multi-step task already takes minutes; two seconds of additional latency per turn is irrelevant, and capability is everything. Which is exactly why the hybrid split lands where it does.

What to write in your context file

Whatever you decide, encode the boundary where it will be read:

## Data handling
- This repository processes customer PII. Do not send file contents
  from `src/pii/` or `fixtures/customer/` to any hosted model.
- Local-model sessions only for anything under those paths.
- Synthetic fixtures in `fixtures/synthetic/` are safe to send.

Then, as with every rule of consequence in this series, back it with something mechanical — a pre-commit hook, a wrapper that refuses, a repository-level configuration that pins the model. The context file makes compliance likely; the mechanism makes violation impossible. Neither substitutes for the other.