Governing Enterprise AI Traffic at Scale: Design and Operation of a Policy-Driven, Multi-Vendor LLM Gateway on Envoy

What happened

PayPal published the design and production record of the Cosmos LLM Gateway, the single governed entry point for all its LLM traffic. As of September 2026 it serves more than 12,000 developers and hundreds of business use cases, including real-time voice support with over 2,000 concurrent calls. It handles millions of requests and about 200 billion tokens a day across 100+ models from several vendors, at 99.999% availability.

1. Two tiers on Envoy. Slow, I/O-bound control work (authentication, credential brokering, policy and budget checks) runs out of process through Envoy ext_proc. Latency-sensitive payload work (wire-format translation, token metering, prompt-cache shaping) runs inside the proxy as a Rust WebAssembly filter, so streamed response chunks never wait on a network call.
2. One internal format. Requests are translated through a canonical intermediate representation, so supporting a new API dialect needs one codec instead of a converter for every pair of formats. Clients can speak OpenAI, Anthropic Messages or Gemini style, and changing the vendor behind a model is a gateway config change.
3. Brokered credentials. Clients hold only internal tokens. The gateway holds vendor keys, attributes every call to a user or app, and never returns the keys.
4. Policy as code. Model access, business-unit restrictions and budgets are declarative policies evaluated by Open Policy Agent, with approvals, scheduled activation, pilots by cohort, shadow mode before enforcement and versioned bundles with last-known-good fallback.
5. Budgets as views. Every spend scope is a view over per-model usage counters, so a new budget instantly covers spend already recorded in its window, with no data migration.
6. Optional DLP. A layer of pattern rules plus an in-house detection model can block, redact or reversibly pseudonymize sensitive data before it leaves for a vendor.
7. Overhead. In a benchmark against a synthetic upstream, the core path adds a median of 4 ms and a p99 of 11 ms at over 2,200 requests per second on two 4-vCPU instances, with no failed requests.

Why it works

Direct integration fails at scale in four ways: keys spread everywhere, code locked to one vendor's format, spend nobody controls, and governance that differs per team. Putting one layer in front fixes all four, but only if it is fast enough for streaming.

The key choice is splitting work by latency profile. Policy decisions can tolerate a network hop once per request. Per-chunk work on a token stream cannot, so it stays in process. Explicit failure rules complete the design: access control and DLP fail closed, observability fails open, and a bad config stops startup instead of serving wrong answers.

The bigger pattern

This is the API gateway and service mesh pattern applied to LLMs, plus FinOps. Other good habits from platform engineering show up too: configuration as validated data instead of deploys, shadow mode before enforcement, and modeling budgets as queries over raw counters rather than separate state that needs migrations.

The new part is LLM-specific: long-lived streaming and voice sessions (up to 30 minutes) that must survive deploys, prompt-cache shaping at the gateway, and token-based metering.

What it means in practice

• If several teams call LLM vendors directly, centralize keys and metering early. Retrofitting attribution is painful.
• Keep per-chunk streaming work in process; push slow decisions out of process, once per request.
• Store usage as raw counters keyed by immutable model IDs, and derive budgets as views. It makes new reports and limits free.
• Write down fail-open and fail-closed rules per component before an incident forces the choice.
• Roll out new policies in shadow mode and compare decisions before you enforce them.
• Plan connection draining for long-lived sessions, the same way you would for WebSockets or video streams.