AI Gateways and Data Retention: Enforcing Policy in the Request Path

Most organisations that believe they need self-hosted inference need a zero-retention contract. The difference is a negotiation rather than an infrastructure project.

What zero data retention actually means

The phrase gets used loosely, so it is worth pinning down. Zero data retention is a contractual arrangement in which a provider does not persist the content of your requests or responses beyond the time needed to serve them. Prompts are not stored, completions are not stored, and neither is available for training or for human review.

It does not mean the data never leaves your network — it does, that is what an API call is. It means it is not kept. Those are different guarantees and teams conflate them constantly, usually in the direction of assuming the stronger one.

Most organisations that believe they need row five actually need row three, and the difference between them is a contract negotiation rather than an infrastructure project. Establishing which row you are on, and which you require, is worth doing before designing anything.

The gateway pattern

The architectural move that makes any of this enforceable is putting a gateway between your developers and the model provider. Every request goes through infrastructure you control.

agent → gateway (yours) → provider

The gateway is where you can:
  authenticate the caller           (which team, which service)
  attribute cost                    (per team, per repository)
  enforce routing rules             (sensitive repos → local model)
  redact before egress              (secrets, PII patterns)
  log requests                      (metadata; content only if you must)
  apply rate and spend limits       (per caller, per day)
  fail over between providers       (availability)

Without a gateway, each of those is a per-developer configuration item, and per-developer configuration is not a control. The gateway converts policy into something that holds by construction.

Routing by data classification

The highest-value rule a gateway enforces is that some content never reaches a hosted provider at all.

routes:
  - name: regulated
    match:
      repository: ["core-banking", "claims-processing"]
    provider: local-cluster
    on_unavailable: fail          # never silently fall back to hosted

  - name: standard
    match:
      repository: ["*"]
    provider: hosted-primary
    fallback: hosted-secondary

redaction:
  - pattern: 'AKIA[0-9A-Z]{16}'         # AWS access key
    action: block                        # refuse the request outright
  - pattern: '-----BEGIN [A-Z ]*PRIVATE KEY-----'
    action: block
  - pattern: '\b\d{3}-\d{2}-\d{4}\b' # SSN shape
    action: redact                       # replace, continue

The on_unavailable: fail line is the one that matters most and the one most likely to be set wrongly. A routing rule that falls back to a hosted provider when the local cluster is down does not enforce anything — it enforces a preference. If the requirement is real, unavailability must mean failure.

Redaction is a mitigation, not a boundary

Pattern-based redaction catches credentials with recognisable shapes: cloud keys, private key headers, tokens with fixed prefixes. It is worth having, and it is worth being clear-eyed about its limits.

• It cannot recognise a password that looks like an ordinary string.
• It cannot recognise proprietary business logic, which is often the thing you actually care about.
• It produces false positives that break legitimate requests, and tuning those away weakens it.

Treat it as a net for the obvious cases rather than a control you can rely on. The control is routing — deciding that certain content does not go to certain providers at all.

Logging without recreating the problem

A gateway that logs full request content becomes a second store of exactly the data you were trying to protect, in a system with weaker access controls than the one it came from.

Metadata answers nearly every operational question — cost, usage, which routes fire, whether redaction triggers. Content answers a small number of incident-response questions and creates a permanent liability. Default to metadata.

Spend control that actually binds

The failure mode nobody plans for is not a breach. It is a bill. An agent in a retry loop, or a scheduled job that ran forty times overnight, consumes budget at machine speed with no natural stopping point.

limits:
  - scope: caller           # per developer or service account
    tokens_per_day: 5_000_000
    on_exceed: reject       # not "warn" — warnings are ignored

  - scope: repository
    tokens_per_day: 20_000_000

  - scope: single_request
    max_input_tokens: 400_000
    max_output_tokens: 16_000

alerts:
  - when: caller_daily > 60%
    notify: the caller, not just the platform team

Two details matter. on_exceed: reject rather than warn, because a warning nobody sees is not a limit. And alerting the caller rather than only the platform team — the person running the session is the one who can stop it, and they usually have no idea their job is looping.

Self-hosting is a bigger commitment than it looks

Running your own inference removes the egress question entirely, and introduces an operational burden that is routinely underestimated: GPU capacity planning, model updates, inference-server upgrades, availability during business hours, and a capability gap on hard tasks that widens whenever hosted frontier models improve.

It is the right answer when egress is genuinely prohibited — some regulated and air-gapped environments — and the wrong answer when the underlying requirement was "our code must not be trained on", which a zero -retention agreement satisfies without any of that.

The useful question is not "hosted or self-hosted" but "what is the actual requirement, and what is the cheapest arrangement that satisfies it". Working that out first is what stops teams building infrastructure to solve a contract problem.