A Gateway Buys You Four Things and Costs You an Outage Surface

When it is down, nobody's agent works. Not degraded, not slower — stopped.

Four things only a gateway gives you

Routing every agent request through a proxy you operate looks like unnecessary infrastructure until you need one of four specific properties, at which point nothing else provides them.

Per-user attribution that survives the provider. Vendor dashboards attribute by whatever identity the vendor sees. Once traffic goes through a cloud provider's hosted deployment, or once several teams share a key, the vendor's view stops being able to answer "who spent this". A gateway sees a per-user credential on the way in and can.

Spend caps that actually stop something. A budget alert tells you afterwards. A gateway can refuse the request that would exceed a limit, per user, per team, per project.

A request log you control. What was sent, by whom, when, how large, to which model, at what cost. Retained on your terms rather than the provider's.

Substitution. One place to change which model serves which traffic, to fail over, or to route a class of request to a cheaper model, without touching a configuration on every developer's machine.

The attribution problem is the usual reason

Worth expanding because it is what drives most gateway adoption. On a cloud-provider deployment, usage is billed to your cloud account and the vendor's own analytics do not cover it. Telemetry export from each developer's machine is the alternative and it depends on every machine being configured correctly, which is a weaker guarantee than a chokepoint everything must pass through.

If you need to answer "which team spent what" with something you would defend in a budget meeting, a gateway is the straightforward way to get it and the alternatives are all approximations.

You are now in the hot path of everything

The cost, stated plainly. Every agent request in the organisation depends on a service you operate. When it is down, nobody's agent works — not degraded, not slower, stopped. When it is slow, every interaction is slower by that amount, and interactive agent work is unusually sensitive to latency because it is conversational.

That is a real operational commitment and it is routinely underestimated, because the gateway is usually stood up by a platform team as a small project and then becomes tier-one infrastructure without ever being reclassified.

Fail open or fail closed, decided in advance

The design question that follows. If the gateway cannot reach the provider, or its own policy store is unavailable, does it refuse the request or pass it through?

Fail closed is correct when the gateway enforces a control you actually depend on — a spend cap, a region restriction, a redaction step. Fail open is correct when it is primarily an observability layer and stopping work is worse than losing a log line. The mistake is not choosing, because the default behaviour of most proxies is to fail closed by timing out, and discovering that during an outage is a poor time to have the debate.

The log is the most sensitive text in your organisation

Here is the consequence people do not anticipate. A gateway that logs request bodies is storing the contents of every repository anyone works on, plus whatever else lands in context: configuration, log excerpts, error messages containing customer data, occasionally a credential someone's agent read before a deny rule caught it.

You built a system whose logs are more sensitive than your source repository, and it was created by a platform team for cost reporting. Treat it accordingly: encrypted at rest, strict access control, a short retention window, and an explicit decision about whether bodies are stored at all.

Redaction before storage, not after

If you do store bodies, redact on the way in. Secret-shaped strings, known personal-data patterns, anything matching your credential formats. It will be imperfect and it removes the most common accidents.

Doing it after the fact does not work: the unredacted copy has already been written, replicated and backed up, and the cleanup is a project rather than a job. The general framing of what leaves your machine and under what terms is in AI gateways and data retention; this is the same question asked about your own infrastructure rather than the vendor's.

What breaks when you insert a proxy

Three things, in rough order of how often they cause confusion.

Prompt caching. Caching depends on stable request prefixes and provider-side cache state. A gateway that normalises requests, reorders fields, or rewrites system content can silently destroy cache hit rates, and the symptom is a large unexplained cost increase with no change in usage. Verify hit rates before and after.

Streaming. Interactive agents stream, and a proxy that buffers turns a responsive session into a series of pauses. It still works, and it feels much worse.

Newer capabilities. A gateway has to understand the request and response shapes it forwards. When the provider adds something — a new content type, a new tool-use field — a strict gateway may drop or reject it until updated, which shows up as a feature that works for some people and not others.

Rate limiting is a fairness mechanism

A worthwhile use beyond spend. Organisation-level provider limits are shared, which means one enthusiastic automated job can consume the capacity everyone else needs. A gateway can allocate that: per-user ceilings, a lower priority for background jobs, a reserved share for interactive work.

That is genuinely difficult to achieve any other way, and it becomes important precisely when agent use is going well, which is when a batch job can plausibly saturate the whole allocation.

Build or buy, and the third option

Open-source proxies exist and cover the common cases well. Building your own is usually a mistake unless you have a requirement they cannot meet, and the requirement is usually integration with an internal identity or policy system rather than anything about the traffic.

The third option is worth mentioning: a first-party gateway from the provider, where one exists, which handles the attribution and spend-limit cases without you operating anything. If your requirement is attribution rather than provider substitution, that is often the better trade — fewer moving parts, and someone else on call for it.

When you do not need one

A small team on subscription plans, where usage is bounded by the plan and attribution is obvious because you can name everyone. A single team with one key and a shared budget nobody is arguing about. Any situation where the answer to "who spent what" is already known.

In those cases a gateway adds an operational dependency to solve a problem you do not have, and the honest recommendation is to wait until one of the four properties at the top of this article becomes something you actually need.

If you are going to run one

Make it properly redundant, because it is now tier-one. Decide fail-open versus fail-closed explicitly and write it down. Log metadata by default and bodies only for a bounded reason. Redact on ingest. Verify cache hit rates and streaming behaviour after every upgrade. Set per-user limits rather than only an organisation-wide one. And review the log retention with whoever owns data policy, because the thing you built is not what the project brief said it was.

Latency compounds in a way batch systems do not

Worth being concrete about the performance cost, because the intuition from ordinary API proxies understates it. An agent session is not one request; it is dozens of sequential model calls, each waiting on the previous. Forty milliseconds added per call is a couple of seconds per session, which is tolerable, and a hundred and fifty milliseconds is not.

So the latency budget for a gateway in this position is tighter than for almost anything else you proxy, and it is worth measuring the added round-trip specifically rather than assuming that a proxy which is fast for web traffic is fast enough here.

Who owns it, and do they know

An organisational hazard specific to this piece of infrastructure. A gateway is usually built by whoever needed the attribution numbers — often a single engineer, often as a side project — and it becomes load-bearing before anyone assigns it an owner or an on-call rotation.

The question to ask early is who is paged when it fails at nine on a Monday. If the answer is "the person who built it, informally", the organisation has a dependency it has not acknowledged, and the time to fix that is before the first outage rather than during the review afterwards.

Start with metadata and one report

A sensible first version, if you decide you need one. Route the traffic, log metadata only, and produce one report: spend per user per week. That answers the question that motivated the project, requires no decisions about body retention, and gives you an operational baseline before anything else depends on it.

Everything else — caps, routing rules, redaction, model substitution — can be added once the thing has proven it stays up. Building all of it at once means the first outage happens with the most complexity and the least operational experience, which is exactly backwards.

The question that decides it

Before any of this: can you already answer, to the standard you need, who spent what and whether anyone can exceed a limit? If yes, a gateway is infrastructure in search of a requirement. If no, and the answer matters to someone with a budget, it is the only mechanism that gives you a definitive answer rather than an estimate assembled from several partial views.

That framing keeps the decision on the property you need rather than on the general appeal of having a chokepoint, which is how these get built and then resented.

The migration nobody plans for

One practical warning if you adopt a gateway. Getting traffic onto it is easy; getting it off is not, because every machine, CI job and integration has been configured to point at it, and several of them are owned by people who have left.

Keep the client-side configuration in one managed place rather than in individual developer settings, so redirecting traffic is a change you make once. That costs nothing at setup time and is the difference between a provider migration taking an afternoon and taking a quarter.

What it changes about developer experience

A consideration that rarely makes the design document. Developers notice a gateway in three ways: when it is slow, when it breaks, and when it blocks them. All three are negative, and the benefits — attribution, caps, logs — accrue to somebody else entirely.

That asymmetry is worth managing deliberately. Make refusals explain themselves, keep the added latency measured and published, and give developers something back — visibility into their own usage is the obvious candidate, and it costs nothing given you are already collecting it. A control that only ever takes things away gets routed around eventually.

Weigh it against the alternative honestly, too. Telemetry exported from each developer's machine gives you most of the attribution without a chokepoint, and it fails differently: it is incomplete rather than unavailable, and a machine that is misconfigured produces a gap in a report rather than stopping someone's work. For teams whose requirement is visibility rather than enforcement, that is frequently the better shape, and it is worth ruling out before committing to run a tier-one service.