← Back to summary
Governing Enterprise AI Traffic at Scale: Design and Operation of a Policy-Driven, Multi-Vendor LLM Gateway on Envoy
Converted from PDF with markitdown. Layout may be rough.
| | | | Governing | | Enterprise | | AI | Traffic | at Scale: | | | | |
| --- | --- | --- | --------- | --- | ---------- | --- | --- | ------- | --------- | --- | --- | --- | --- |
DesignandOperationofaPolicy-Driven,Multi-VendorLLMGatewayonEnvoy
| | | MithunSasidharan* | | | SrinivasanManoharan | | | | RupeshThakur | | | | |
| --- | --- | ----------------- | --------- | --- | ------------------- | ----------- | --- | ----------- | ------------ | --- | --- | --- | --- |
| | | | AbhinavTK | | | AshwinHegde | | AnandGautam | | | | | |
PayPalInc.
Abstract
6202 tcO 62 ]CD.sc[ 1v90553.0162:viXra requestspersecondontwo4-vCPUinstances,withnofailed
| | | | | | | | requestsacrossthetestedrange. | | | | Wereportthearchitecture, | | |
| ------------------- | ------- | ------- | ------- | ----- | -------- | ------ | ----------------------------- | ------ | ------------------------ | --- | ------------------------ | ------- | ----------- |
| | | | | | | | the | policy | model, an evaluation, | | and the | lessons | we would |
| Large organizations | | now | consume | large | language | models | | | | | | | |
| | | | | | | | carry | to any | enterprise standardizing | | access | to | multiple AI |
| (LLMs) from | several | vendors | at | once, | and each | vendor | | | | | | | |
providers.
bringsitsownAPIdialect,credentialmodel,streamingpro-
tocol, pricing and release cadence. Integrated directly, this Keywords: LLMgateway,AIgovernance,Envoy,ext_proc,
heterogeneity yields credential sprawl, format lock-in, un- WebAssembly,policyascode,OpenPolicyAgent,dataloss
boundedspendandinconsistentgovernance. Wepresentthe prevention,FinOps,multi-vendorrouting,observability.
designandproductionoperationoftheCosmosLLMGate-
way, the governed entry point for enterprise LLM traffic 1 Introduction
| at PayPal. | The | gateway is | built on | Envoy | and splits | work | | | | | | | |
| ---------- | -------- | ---------- | -------- | --------- | ---------- | ---- | ---------- | --- | ----------------- | --- | ---------- | ---------- | --- |
| | | | | | | | Generative | | AI quickly became | a | production | dependency | at |
| by latency | profile: | I/O-bound | control | decisions | (authenti- | | | | | | | | |
cation, credential brokering, policy and budget evaluation) PayPal. Engineers use agentic coding tools daily; product
run in an out-of-process external-processing service, while teams ship AI-assisted features; and customer-facing sys-
latency-sensitive payload work (wire-format translation, to- tems, including real-time voice support, depend on frontier
kenmeteringandprompt-cacheshaping)runsin-proxyasa modelsbeingavailable,governedandaffordable. Themod-
elstheseworkloadsneedcomefromseveralvendorssimul-
| Rust WebAssembly | | filter, | so that, | by default, | streamed | re- | | | | | | | |
| ---------------- | --- | ------- | -------- | ----------- | -------- | --- | --- | --- | --- | --- | --- | --- | --- |
sponse body chunks never wait on an out-of-process call. taneously, and the best model for a task can change on a
Models, providers and prices are largely a validated data timescaleofweeks.
model rather than code. Governance is declarative policy Theobviouswaytoconsumethesemodelsisforeachap-
evaluated by Open Policy Agent and managed through a plicationtointegrateeachvendordirectly: holdthevendor’s
| | | | | | | | key, | adopt | its request format, | handle | its | streaming | seman- |
| ------------- | ---- | ----------- | --------- | --- | ------------ | --- | ---- | ----- | ------------------- | ------ | --- | --------- | ------ |
| control plane | with | role-scoped | authoring | | for platform | and | | | | | | | |
business-unitadministrators,approvalworkflows,scheduled tics, and track its spend locally. That approach works for
activation, limited-access model releases, cohort-based pi- oneteamandonemodel. Itdoesnotsurviveanorganization
lots, shadow-mode rollout, configurable usage notifications ofthousandsofengineersandhundredsofproductionwork-
andversionedpolicy-bundledistribution. Forusecasesthat loads spanning several major providers. Direct integration
failsalongfourcompoundingaxes:
requireit,adata-loss-preventionlayerthatcombinespattern
rules with an internally hosted detection model can block, • Credential sprawl. Long-lived vendor keys proliferate
redactorreversiblypseudonymizedetectedsensitivecontent acrossapplications,withnocentralplacetoscope,rotate,
before it reaches a provider. Every spend scope is a view attributeorrevokethem,andnoreliablelinkfromacallto
over per-model usage counters, so a new budget applies to thehumanwhoauthorizedit.
spend already recorded in its window without data migra- • Format lock-in. An application written against one ven-
tion, and every governed request is designed to yield one dor’s API cannot adopt another vendor’s model without
deduplicatedusagerecordintheanalyticswarehouse. Asof code changes, so model choice, regional placement and
September2026,thegatewayservesmorethan12,000devel- failoverbecomeengineeringprojectsratherthanconfigu-
| opersandhundredsofbusinessusecases,includingreal-time | | | | | | | ration. | | | | | | |
| ----------------------------------------------------- | --- | --- | --- | --- | --- | --- | ------- | --- | --- | --- | --- | --- | --- |
voicesupportwithmorethan2,000concurrentcalls. Ithan- • Ungovernedspend. Costaccruesperkeyandpervendor,
dlesmillionsofrequestsandroughly200billiontokensper withnoorganization-awarebudgets,noearlywarningand
dayacrossmorethan100modelsfrommultipleproviders,at noleverforbusiness-unitleadership.
99.999%availabilitytobusiness. Inabenchmarkofthecore • Uneven governance and visibility. Each integration au-
request path against a synthetic upstream, it adds a median thenticates, logs, rate-limits and protects data differently;
of 4ms and a p99 of 11ms of overhead at more than 2,200 securityandfinancehavenosingleplaneonwhichtoob-
1
serveorcontrolAItraffic. Embeddings, Anthropic Messages, and Gemini generate-
Thispaperdescribeshowweaddressedthesefailuremodes Content)andreachapprovedmodelsthroughtheconversion
behindasinglelayer,theCosmosLLMGateway. Beyonda pathstheregistrydeclares(table3). Changingthevendoror
unifiedAPIanddataplane,thesystemcomprisesafullgov- regionbehindamodelisagatewaychange, notanapplica-
| ernance control | | plane, a | policy-as-code | | decision | engine, | a | tionchange. | | | | | | |
| --------------- | --- | -------- | -------------- | --- | -------- | ------- | --- | ----------- | --- | --- | --- | --- | --- | --- |
platform-identityserviceattestedbycorporatesinglesign-on R2 Brokered credentials. Clients present only an inter-
| (SSO), a | durable | usage-attribution | | pipeline | and | inline | data- | | | | | | | |
| -------- | ------- | ----------------- | --- | -------- | --- | ------ | ----- | --------------- | ---------- | ------ | ----- | --- | ----------- | --- |
| | | | | | | | | nal credential: | a personal | access | token | for | developers, | an |
lossprevention(DLP).Thispaperdocumentsthesystemas application credential bound to a registered application for
itnowoperates.
businessapplications,oraplatform-mintedtoken(section8).
Contributions. Wemakethefollowingcontributions: The gateway obtains the upstream vendor credential, at-
| | | | | | | | | tributes | its use to the | calling | user or application, | | and | never |
| --- | --- | --- | --- | --- | --- | --- | --- | -------- | -------------- | ------- | -------------------- | --- | --- | ----- |
1. Atwo-tierEnvoyarchitecturethatrunsI/O-boundgov-
ernance out of process and streaming translation in pro- returnsittotheclient.
| cess, joined | by | an explicit, | contract-tested | | handoff | | (sec- | | | | | | | |
| ------------ | --- | ------------ | --------------- | --- | ------- | --- | ----- | ----------------------- | --- | --- | -------------------------- | --- | --- | --- |
| | | | | | | | | R3Identity-awarepolicy. | | | Everyrequestmadeonbehalfof | | | |
tion5). apersonischeckedagainstthecaller’scurrentpositioninthe
| 2. Models, | routing | and | pricing | as validated | | data: | a reg- | | | | | | | |
| ---------- | ------- | --- | ------- | ------------ | --- | ----- | ------ | -------------- | ---------- | --- | ------------------ | --- | --- | ------- |
| | | | | | | | | organizational | hierarchy, | so | model restrictions | | and | budgets |
istrywhoseloaderenforcesaserviceabilityinvariant,and followpeoplethroughreorganizations.
| a control-plane | | onboarding | lifecycle | | that takes | a model | | | | | | | | |
| --------------- | --- | ---------- | --------- | --- | ---------- | ------- | --- | ------------------------------ | --- | --- | --- | ----------------- | --- | --- |
| | | | | | | | | R4Configuration,notdeployment. | | | | Onboardingamodel, | | |
fromstagedtogenerallyavailablewithoutadeployment
changingaprice,oraddingabudget,restrictionornotifica-
(sections6and10.1).
tionisavalidateddatachange.
3. Acanonicalintermediate-representation(IR)transla-
tion layer R5 Streaming-safe performance. Translation and meter-
| | that | needs | one codec | per wire | format | instead | | | | | | | | |
| --- | ---- | ----- | --------- | -------- | ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- |
ingmustnotbufferstreamedresponsesoraddmaterialtime-
| of one | converter | per format | pair, | with | fast paths | that | are | | | | | | | |
| ------ | --------- | ---------- | ----- | ---- | ---------- | ---- | --- | --- | --- | --- | --- | --- | --- | --- |
losslessforclientcontentandgateway-sideprompt-cache to-first-token. Gateway overhead must be small and stable
underload.
shaping(section7).
4. Alayeredaccessandspendpolicymodel:organization- R6 Data protection. Where a use case’s data classifica-
| anchored | model | restrictions, | | business-unit | controls, | | and | | | | | | | |
| -------- | ----- | ------------- | --- | ------------- | --------- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
tionrequiresit,thegatewayappliesdetectiontopromptsand
budgetswhosescopesareviewsoverper-modelcounters blocks,redactsorpseudonymizesdetectedpersonalandpay-
keyedbyimmutablemodelidentifiers,whichmakesthem mentdatabeforeitreachesanexternalprovider,withpolicy
| retroactivewithinawindowandmigration-free. | | | | | | Policies | | setperusecase. | | | | | | |
| ------------------------------------------ | --- | --- | --- | --- | --- | -------- | --- | -------------- | --- | --- | --- | --- | --- | --- |
areevaluatedbyapuredecisionfunctionseparatedfrom
| | | | | | | | | R7 Explicit | failure | semantics. | Access | control | and | data |
| --- | --- | --- | --- | --- | --- | --- | --- | ----------- | ------- | ---------- | ------ | ------- | --- | ---- |
sideeffects(section9).
| | | | | | | | | protection | fail closed; | observability | fails | open; | an | invalid |
| ------------- | --- | ---------------- | --- | ------------ | --- | --------- | --- | ---------- | ------------ | ------------- | ----- | ----- | --- | ------- |
| 5. A governed | | policy lifecycle | | for platform | and | business- | | | | | | | | |
configurationpreventsstartupinsteadofservingincorrectly.
| unit administrators: | | approvals, | | scheduled | activation, | | pi- | | | | | | | |
| -------------------- | ------------- | ---------- | ----------------------- | --------- | ----------- | ------------ | --- | ------------------------------ | ------------ | -------- | ----------- | ------------------- | ------- | --- |
| | | | | | | | | R8Connections,notonlyrequests. | | | | Long-livedreal-time | | |
| lots with | tier | ladders, | shadow-to-authoritative | | | rollout, | | | | | | | | |
| | | | | | | | | sessions | (voice calls | of up to | 30 minutes) | must | survive | de- |
| threshold | notifications | | and versioned | | bundle | distribution | | | | | | | | |
ployments,autoscalingandnodeupgrades.
withlast-known-goodsemantics(section10).
| | | | | | | | | R9Attributionbydefault. | | | Everygovernedrequestisde- | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | ----------------------- | --- | --- | ------------------------- | --- | --- | --- |
6. Adurable,priced,per-requestusagerecordthatdrives
chargebackandanalytics,alongsidetheper-modelcoun- signedtoproduceadurable,pricedusagerecordattributable
toitscallingprincipal(auseroraregisteredapplication)and,
tersthatdrivebudgets(section11),togetherwithproduc-
tionevidence,anoverheadbenchmarkandthelessons wherepresent,toaplatformandworkflow,withnoapplica-
| wewouldcarrytosimilarsystems(sections13and14). | | | | | | | | tioninstrumentation. | | | | | | |
| ---------------------------------------------- | --- | --- | --- | --- | --- | --- | --- | -------------------- | --- | --- | --- | --- | --- | --- |
The gateway governs which models an AI client may call. R10 No big-bang migration. Hundreds of applications
Acompanionplatform,theCosmos.AIMCPHub[23],gov- were already in production. Any plan that required every
ernswhichtoolsitmayreachovertheModelContextProto- teamtoupdateitsintegrationwasnotacceptable.
col[14]. Togethertheyformtheaccessandcontrolplanefor
| enterpriseAIatPayPal. | | | | | | | | 3 Background | | and | Related | Work | | |
| --------------------- | --- | --- | --- | --- | --- | --- | --- | ------------ | --- | --- | ------- | ---- | --- | --- |
2 Requirements Programmable proxies. Envoy [6] is a widely deployed
| | | | | | | | | layer-7 (L7) | proxy | with mature | TLS, | HTTP/2, | WebSocket, | |
| ---------- | --- | ------------ | ---- | ----------- | ----- | --- | ----- | -------------- | ----- | ----------- | -------- | --------- | ---------- | ------ |
| We derived | the | requirements | from | the failure | modes | | above | | | | | | | |
| | | | | | | | | load-balancing | and | telemetry | support. | It offers | two | exten- |
and from the operating constraints of a regulated financial- sion mechanisms that matter here. The external processing
servicescompany.
| | | | | | | | | filter (ext_proc) | [7] | streams | request and | response | phases | to |
| --- | --- | --- | --- | --- | --- | --- | --- | ----------------- | --- | ------- | ----------- | -------- | ------ | --- |
R1 One API, approved models. Clients use APIs they agRPCservicethatmayinspectandmutateheaders,bodies
already know (OpenAI Chat Completions, Responses and and trailers and may return an immediate response; it runs
2
outofprocessandcanbeconfiguredtofailopenorclosed. Setting controlext_proc metricsext_proc
| Proxy-Wasm | [21] | runs | sandboxed | WebAssembly | | modules | | | | | | | | |
| ---------- | ---- | ---- | --------- | ----------- | --- | ------- | --- | ----------- | --- | ---------- | --- | -------- | --- | --- |
| | | | | | | | | Failuremode | | failclosed | | failopen | | |
insidetheproxy’sworkerthreads,whichsuitsper-bytework Requestheaders send send
| onstreamingbodies. | | | | | | | | Requestbody | | buffered | | streamed(mirrored) | | |
| ------------------ | --- | --------------------------------- | --- | --- | --- | --- | --- | --------------- | --- | -------- | --- | ------------------ | --- | --- |
| | | | | | | | | Responseheaders | | skip | | send | | |
| LLMgateways. | | Open-sourceLLMproxiesprovideauni- | | | | | | | | | | | | |
| | | | | | | | | Responsebody | | none | | streamed(mirrored) | | |
fiedAPIacrossvendors,typicallyasanapplication-levelser-
| | | | | | | | | Responsetrailers | | send | | send | | |
| ---------------- | ------- | --------- | ----------- | ------ | --------- | -------- | ------ | --------------------------------------- | --- | ---------------- | --- | ----------------------- | --- | --- |
| | | | | | | | | Blocking | | yes | | no(observabilitymode) | | |
| vice [20]. | Agent | Router | (formerly | Envoy | AI | Gateway) | [1] | | | | | | | |
| | | | | | | | | Dynamicmetadata | | attribution,cost | | – | | |
| and agentgateway | | [2] build | AI-specific | | routing | on | top of | | | | | | | |
| proxy data | planes, | and cloud | providers | | now offer | managed | | | | | | | | |
| | | | | | | | | Table1: Defaultext_procprocessingmodes. | | | | Responsebodiesneverwait | | |
| AI gateways | [5, | 13]. | The closest | design | to | ours, | Agent | | | | | | | |
onanout-of-processcallunlessausecase’sdata-loss-preventionpolicyre-
| Router, | performs | request | translation, | | upstream | authoriza- | | | | | | | | |
| ------- | -------- | ------- | ------------ | --- | -------- | ---------- | --- | --- | --- | --- | --- | --- | --- | --- |
quiresresponsehandling(section8).
tionandtokenmeteringinanexternalprocessor,andnames
Envoy dynamic modules as a possible higher-performance control plane holds the authoritative model registry, prices
andgovernancepolicy,exposesamanagementAPIforplat-
| alternative | [1]. | A single | streamed | response | can | carry | thou- | | | | | | | |
| ----------- | ------- | -------- | -------- | -------- | ------ | --------- | ----- | -------- | ------------- | ---- | --------------- | --- | --- | --------- |
| | | | | | | | | form and | business-unit | (BU) | administrators, | | and | publishes |
| sands of | chunks, | and we | did | not want | to pay | a network | | | | | | | | |
roundtriptoanexternalprocessforeachofthem. Wethere- versionedpolicybundles.Theidentityplanemintsverifiable
| | | | | | | | | JSON Web | Tokens | (JWTs) | for | platforms | that call | the gate- |
| --------- | ---- | --------- | --------- | --- | ---------- | --- | ------- | -------- | ------ | ------ | --- | --------- | --------- | --------- |
| fore keep | only | I/O-bound | decisions | out | of process | | and run | | | | | | | |
per-chunktranslationandmeteringin-proxyasWebAssem- wayonbehalfofusers. Thetelemetryplanecarriesmetrics,
bly (section 5). We evaluated an off-the-shelf proxy before tracesanddurableper-requestusagerecordsintotheanalyt-
icswarehouse.
| building. | Basic | routing | worked, | but at | the time | of | evalu- | | | | | | | |
| --------- | ----- | ------- | ------- | ------ | -------- | --- | ------ | --- | --- | --- | --- | --- | --- | --- |
ation it did not meet our requirements for real-time (Web- Deployment. The gateway runs on Google Kubernetes
Socket) traffic, identity-aware authorization, enforceable Engine (GKE) in multiple regions with disaster recov-
user- and organization-level budgets, or workload-identity- ery. Closed-weight models such as Gemini and Claude are
| basedcloudcredentials. | | | Ourdesignsharestheproxy-native | | | | | | | | | | | |
| ---------------------- | --- | --- | ------------------------------ | --- | --- | --- | --- | ------- | ----- | ---------- | ------- | ---------- | -------- | --- |
| | | | | | | | | served, | where | supported, | through | the global | endpoint | of |
approach of these projects. It differs mainly in the gover- Google’s Gemini Enterprise Agent Platform (formerly Ver-
nancelayer: organization-anchoredpolicy,retroactivespend tex AI Platform), which avoids dependence on a single re-
scopes,andamanagedpolicylifecycle. gional endpoint. Open-weight models run on an internal
GKEinferencestackbuiltonvLLM[12],SGLang[24]and
| Authorization | | and policy | as | code. | Zanzibar | [19] | es- | | | | | | | |
| ------------- | --- | ---------- | --- | ----- | -------- | ---- | --- | --- | --- | --- | --- | --- | --- | --- |
tablished relationship-based authorization at global scale. NVIDIADynamo[15].Otherprovidersarereachedoverthe
OpenPolicyAgent(OPA)[17]popularizeddeclarative,side- networkwithbrokeredcredentials. Thecallerdoesnotneed
toknowwhereamodelruns.
| effect-free | policy | evaluated | against | injected | data. | | We use | | | | | | | |
| ----------- | ------------ | --------- | ------- | --------- | ------- | -------- | ------ | ------ | ------ | -------- | --- | ------------- | --- | --- |
| OPA as | the decision | engine. | The | gateway | gathers | state | and | | | | | | | |
| | | | | | | | | 5 Data | Plane: | Two-Tier | | Extensibility | | |
| performs | effects, | which | keeps | decisions | pure, | testable | and | | | | | | | |
replayable(section9).
| | | | | | | | | 5.1 Splittingworkbylatencyprofile | | | | | | |
| ---------------------------------- | --------- | ------------- | ------------ | ----------- | ---------------- | -------- | ------- | --------------------------------- | ------------- | -------------- | ---------- | ------------------- | ------------- | --------- |
| Cost-aware | model | use | and serving. | | FrugalGPT | | [4] and | | | | | | | |
| | | | | | | | | The most | consequential | | design | decision | was to refuse | the |
| RouteLLM | [16] | route queries | | across | models | to trade | cost | | | | | | | |
| | | | | | | | | trade-offbetweengoverned | | | andfast. | Governancedecisions | | |
| against | quality. | Serving | systems | such | as vLLM | [12] | and | | | | | | | |
| | | | | | | | | are I/O-bound: | | they need | identity | validation, | the | organiza- |
| SGLang[24]optimizeinferenceitself. | | | | | Theselinesofwork | | | | | | | | | |
| | | | | | | | | tional hierarchy, | | spend counters | | and secret | material. | They |
| complement | ours: | a gateway | is | the natural | place | to | enforce | | | | | | | |
| | | | | | | | | therefore | run out | of process | in | a Go ext_proc | service | that |
| the budget | decisions | that | a router | optimizes | for, | and | it sits | | | | | | | |
| | | | | | | | | scales independently | | of | the proxy. | Payload | work | is CPU- |
abovetheservingsystemsthathostopen-weightmodels.
| | | | | | | | | bound, sits | on | the critical | path | of every streamed | | byte, and |
| -------- | --- | -------- | --- | --- | --- | --- | --- | ----------- | ------- | ------------ | --------- | ----------------- | ------------ | --------- |
| | | | | | | | | needs no | network | I/O. It | therefore | runs | inside Envoy | as a |
| 4 System | | Overview | | | | | | | | | | | | |
RustProxy-Wasmfilter,sothatastreamedresponseistrans-
Figure1showsthesystem. AsingleTLSlistenerterminates lated and metered chunk by chunk without waiting on any
alltrafficandroutesitbypaththroughthesameEnvoypod. out-of-processcall.
UnifiedAPIrequestspassthroughthecontrolpath(authen- Below, the control service is the Go service behind the
tication, policy, data protection and credential brokering), control ext_proc, the metrics service is the service behind
translationandmeteringintheWasmfilter,andanupstream the metrics ext_proc, and the Wasm filter is the in-process
call with a brokered credential (section 5). Real-time voice translation filter (translate.wasm in fig. 1). The HTTP filter
APIsrunoverWebSocket;forthese,Envoyholdseachlong- chain runs in a fixed order: (1) the control ext_proc, con-
livedconnectiononitseventloopandforwardsaudioframes figured to fail closed, which receives the buffered request
astheyarrive,withoutbodybuffering(section12). body because the routing key (model) is in the body; (2)
Behind the data plane sit three supporting planes. The a metrics ext_proc, configured to fail open, which runs af-
3
PayPalprivatenetwork
| | | | Envoydataplane(autoscaled) | | | | Control&identityplanes | |
| --- | ------- | --- | -------------------------- | --- | --- | ---------------------- | ---------------------- | --- |
| | Clients | | | | | Externalmodelproviders | | |
OpenAI/Anthropic/GeminiSDKs
| | | | | | GeminiEnterpriseAgentPlatform | | ManagementAPI | |
| ----------------- | --- | --- | ----------------------- | --- | ----------------------------- | ------ | ---------------------- | --- |
| Codingagents&CLIs | | | TLSlistener+pathrouting | | | | | |
| | | | | | Gemini·Claude(globalendpoint) | | platform&BUadmins | |
| | | | controlext_proc | | | OpenAI | Registry,prices&policy | |
Agenticworkflows
| | | | auth·policy·DLP·credentials | | | | durablesystemofrecord | |
| --- | --- | --- | --------------------------- | --- | --- | --- | --------------------- | --- |
fail-closed
xAI
| | | | metricsext_proc | | | | Projector+bundlepublisher | |
| --- | --- | --- | --------------- | --- | --- | --- | ------------------------- | --- |
Businessapplications
| | | | observabilitymode·fail-open | | | | versioned,leader-elected | |
| --- | --- | --- | --------------------------- | --- | --- | --- | ------------------------ | --- |
Azure-hostedmodels
| | | | translate.wasm(Rust) | | | | Platformtokenmint | |
| -------------- | --- | --- | ---------------------------- | --- | --------------------------- | ------------- | ----------------- | --- |
| Real-timevoice | | | | | | policybundles | | |
| | | | in-proxytranslation·metering | | | | SSO→signedJWT | |
| | | | router+upstreamclusters | | Self-hostedopen-weight(GKE) | | | |
vLLM·SGLang·NVIDIADynamo
Telemetryplane:metrics·traces·per-requestusagerecords→warehouse
Figure1: Architecture. OnegovernedentrypointfrontsenterpriseLLMtraffic. OntheunifiedAPI,thecontrolservicemakesI/O-bounddecisionsoutof
processandfailsclosed,andtheWasmfiltertranslatesandmetersinprocess.Real-timesessionsuseaconnection-orientedpathtunedforlong-livedstreams.
Thecontrolplanepublishesversionedpolicybundlesthatthedataplaneenforcesoneveryrequest.TheshadedregionisPayPal’sprivatenetwork:self-hosted
open-weightmodelsruninsideit,andexternalprovidersarereachedwithbrokeredcredentials.
ter control and therefore observes the request as authorized format,tapstokenusagefromthestream,andemitsusageas
and rewritten; (3) the translation Wasm filter; (4) a small response trailers. In the response-trailers phase the control
bufferfilterthatrestoresContent-Lengthaftertranslation, service prices the request from the registry, records spend,
because some upstreams reject chunked request bodies and attaches cost metadata, and invalidates a cached upstream
Proxy-Wasmcannotsetheadersduringthebodyphase;and credentialifthevendorreturned401or403.
| (5) the router. | Table | 1 lists the | ext_proc processing | modes. | | | | |
| --------------- | ----- | ----------- | ------------------- | ------ | --- | --- | --- | --- |
5.3 Acontractbetweentwolanguages
| The metrics | service runs | in Envoy’s | observability | mode, in | | | | |
| ----------- | ------------ | ---------- | ------------- | -------- | --- | --- | --- | --- |
which bodies are mirrored to it asynchronously and never TheGocontrolserviceandtheRustfiltercommunicateonly
held.Thecontrolservicebuffersonlytherequestbody,skips through a small, versioned contract of gateway-internal re-
| response | headers and | bodies entirely, | and is invoked | again | | | | |
| -------- | ----------- | ---------------- | -------------- | ----- | ----- | --------------------- | --------------- | --------------- |
| | | | | | quest | metadata and response | trailers, which | carry the entry |
onlyforresponsetrailers, whereitreadsusage. Bydefault, and upstream wire formats, the upstream model, token us-
the streamed response body therefore sits on the blocking age, a count of unclassified fields and the tools the model
critical path of exactly one extension, the in-process filter; invoked. Thecontractlivesinonemachine-readablefilethat
themetricsserviceseesanon-blockingmirror,andthecon- alsonamestheaccess-logmetadatafields(section11). Test
trolserviceisconsultedagainonlyforthetrailersattheend
| | | | | | suites | in both languages | assert their constants | against it, so |
| --- | --- | --- | --- | --- | ------ | ----------------- | ---------------------- | -------------- |
ofthestream.Inproduction,themetricsservicederivesmet- anydriftfailsthebuild, andanunrecognizedvalueisnever
ricsfrommirroredbodieswithoutpersistingtheircontent. reinterpreted. In practice, this explicit cross-language con-
tractremovedaclassofintegrationbugsthatotherwisesur-
5.2 Requestlifecycle
facedonlyinend-to-endtests.
| Figure 2 | traces a unified | request. | In the request-headers | | | | | |
| -------- | ---------------- | -------- | ---------------------- | --- | --- | --- | --- | --- |
5.4 Failuresemantics
phasethecontrolservicerecordstheentrywireformat(from
thepath)andthecaller’scredential;Envoythendeliversthe Each component on the request path had to answer one
buffered,size-boundedbody.Intherequest-bodyphaseitau- question: if this breaks, what happens to the request? Ta-
thenticatesthecaller,extractsthemodel,resolvesitthrough ble2summarizestheanswers. Theasymmetryisdeliberate.
the registry, evaluates model access, brokers the upstream An access or data-protection rule that silently stops enforc-
credential, evaluates rate and spend policy, sets the handoff ing during an outage is not a rule, so those decisions fail
headersfortheWasmfilter,rewritesthepath,andasksEnvoy closed. Atelemetryoutagemustneverbecomeatrafficout-
to clear its route cache so that Envoy re-selects the vendor age,soobservabilityfailsopen. Newbudgetsarerolledout
route and cluster for the rewritten request. It also attaches inshadowmodefirst, whichisdesignedtokeepadefectin
attributionmetadatafortheaccesslog. TheWasmfilterthen anewlyintroducedpolicyfromtakingAItoolingofflinefor
translates the request into the upstream format. On the re- thewholeorganization. Togethertheserulesmakedegrada-
sponse,thefiltertranslateseachchunkbackintotheclient’s tiongracefulandpredictable: telemetryfailuresneverreach
4
Client Envoy Control Wasm Vendor Failure Behavior
request+ Controlserviceunavailable Requestrejected(failclosed)
internalidentity Access-policyevaluationerror Deny
Spend-policyevaluationerror Deny when enforcing; logged in
headers
shadowmode
bufferedbody
Data-protectioncheckcannotcom- Requestnotsent(failclosed)
1authenticatecaller plete
2resolvemodel(registry) Invalidregistry/policyatboot Processrefusestostart
3modelaccess(OPA) Policybundlereloadfails Keeplast-known-goodrevision
4brokervendorcredential Metricsserviceunavailable Trafficunaffected(failopen)
5rate/spend(OPA)
Clientresetmid-stream Usage record still written (sec-
6handoffhdrs,rewritepath,
clearroutecache tion11)
Table2:Failuresemanticsontherequestpath.
mutations
body
els (with aliases, tags, entry formats and access metadata),
entry→upstreamformat
deployments (physical serving locations with region, prior-
vendorrequest+
brokeredcredential ityandprice), entryroutesandconverters(legaltranslation
stream/unary pairs).
upstream→entry,perchunk; Resolution. For a request with entry path p
tallytokens→trailers
and model identifier x, the data plane computes:
trailers(usage)
entry = entry_routes[p].format
price,spend,attribution model = resolve(x) # canonical id or alias
responsein family = model.upstream_family or family(entry)
client’sformat dep = model.deployment_for(entry)
op = operations[(dep.provider, dep.format, family)]
up_fmt = canonical(family) or dep.format
Figure2: Lifecycleofaunifiedrequest. Routingdecisionshappeninthe require converter(entry -> up_fmt) # identity or declared
bodyphasebecausethemodelisinthebody.Streamedresponsebytespass path = substitute(op.path, dep) # project, region, model,
throughonlythein-processfilteronthecriticalpath. verb
cred = providers[dep.provider].credential
the caller, a failed policy update leaves the last good revi- Capability gating is therefore data. If no responses-family
sioninforce,andaninvalidconfigurationisrejectedbefore operationexistsforaprovider,thatproviderdoesnotsupport
itservestraffic. the Responses API, and no code needs to check for it. An
optional per-model upstream_family pins the vendor
surface when the client’s family would choose the wrong
6 Models, Routing and Pricing as
one. For example, a coding agent may speak a chat-family
Validated Data API while the vendor requires its responses endpoint for
toolusecombinedwithreasoning.
Earlyversionsencodedvendorselection,upstreampathsand
capability flags in code. That made every new model a de-
Pricing as configuration. Each deployment carries its
ployment, and nothing could be edited by an administrator.
price in USD per million tokens for input, output, cache
We replaced it with a registry: a typed, declarative model
reads, cache writes and extended-lifetime cache writes. A
of everything the gateway supports. Model resolution, up-
price may also define a long-context tier that applies when
streampaths,capabilitygatingandpricesallcomefromthe
the prompt (input, cache-read and cache-write tokens) ex-
registry. Thecodekeepsapath-templatesubstitutor,thecre-
ceedsathreshold. Forarequestwithtokencountsn andthe
k
dential resolvers (which perform I/O), and a small number
applicabletier’sratesp
k
,thecostinmicro-dollarsis∑
k
n
k
p
k
.
of provider-specific adaptations for vendor quirks, such as
A price change is therefore a registry edit that takes effect
session-affinityheaders.
withinonepublicationcycle,andthemanagementAPIval-
Entities. The registry defines brands (who makes a idates prices on every write: units come from an allow list,
model), API families (chat, responses, embeddings), wire everyrateisnon-negative,andunknownkeysarerejected.
formats (e.g., Anthropic Messages, OpenAI Chat, OpenAI
Responses, Anthropic-on-Vertex, GeminigenerateContent), Fail-closed validation and the serviceability invariant.
credentials (how a provider authenticates, holding secret The loader turns raw configuration into a typed registry or
names and never values), providers, clusters, provider op- returnsanaggregatederrorthatabortsstartup. Itchecksthat
erations keyed by (provider, deployment format, API fam- every reference resolves, that credential kinds match their
ily) that map to a path template and streaming mode, mod- providers,thatoperationsandaliasesdonotcollide,andthat
5
Upstreamformat Threeconversionplans. BuildingtheIRisnecessaryonly
whentheenvelopecannotsimplyberewritten.Theconverter
| Entryformat | | OAIChat | OAIResp. | | Anth.@Vertex | Gemini | | | | | | | |
| ----------- | --- | ------- | -------- | --- | ------------ | ------ | --- | --- | --- | --- | --- | --- | --- |
choosesamong:
| AnthropicMessages | | | T | T | R | T | | | | | | | |
| ----------------- | --- | --- | --- | --- | --- | --- | ----------- | ----------------------------------------- | --- | --- | --- | --- | --- |
| | | | | | | | • Identity: | sourceequalstarget,andbytespassthroughun- | | | | | |
| OpenAIChat | | | T∗ | T | T | T | | | | | | | |
| OpenAIResponses | | | – | I | – | – | changed. | | | | | | |
GeminigenerateContent – – – I • Raw: same family with a different envelope (e.g.,
| | | | | | | | Anthropic | Messages | | to Anthropic-on-Vertex). | | | The mes- |
| --- | --- | --- | --- | --- | --- | --- | --------- | -------- | --- | ------------------------ | --- | --- | -------- |
Table3:Conversionplanbyentryandupstreamformat:OAI=OpenAI.I=
sagecontentiskeptverbatimandonlyenvelopefieldsare
identity,R=rawenveloperewrite,T=translationthroughtheIR,–=notof-
| | | | | | | | rewritten | (plus | the gateway’s | | cache | breakpoints), | which |
| --- | --- | --- | --- | --- | --- | --- | --------- | ----- | ------------- | --- | ----- | ------------- | ----- |
fered(byregistryconfiguration,notcode).VertexdenotestheGeminiEnter-
priseAgentPlatformAPI.∗Same-formatroundtripthatpreservesGemini
preservesvendor-onlyfieldssuchascachecontrolandex-
tool-callthoughtsignatures(section7). tendedthinkingthatanIRround-tripmightdrop.
| | | | | | | | • Translate: | | a full decode–IR–encode, | | | for | example from |
| --- | --- | --- | --- | --- | --- | --- | ------------ | --- | ------------------------ | --- | --- | --- | ------------ |
thefollowinginvariantholds:
AnthropicMessagestoOpenAIChat.
| | | | | | | | Both fast | paths | are | lossless | for client | content | because |
| ----- | --- | -------------------- | ---------- | --- | -------------- | ------- | ---------- | ------------ | --------- | --------- | ------------ | ------- | -------------- |
| ∀m∈M, | ∀f | ∈entry(m):∃d∈dep(m): | | | | | | | | | | | |
| | | | | | | | they never | build | the | IR. Table | 3 shows | the | plan for the |
| | | | | | (cid:0) | (cid:1) | main entry | and | upstream | pairs. | OpenAI | Chat | to OpenAI |
| | | | op(d,f)̸=⊥ | ∧ | conv f,up(d,f) | . | | | | | | | |
| | | | | | | | Chat is | a deliberate | exception | | to identity. | | Gemini, served |
In words, every format a model advertises is backed by at through an OpenAI-compatible chat endpoint, returns a
thought_signature[9]oneachtoolcall(inavendorex-
| least one | deployment | | that has | both an | operation | and a con- | | | | | | | |
| --------- | ---------- | --- | -------- | ------- | --------- | ---------- | --- | --- | --- | --- | --- | --- | --- |
verter. The model-discovery endpoint is rendered from the tension field, extra_content) and requires it to be sent
| | | | | | | | back on | the next | turn; | some | agent | clients | drop unrecog- |
| ------------------------------------------------- | --- | --- | --- | --- | --- | --- | ------------ | -------- | ----- | -------- | ------------- | ------- | ------------- |
| sameregistry,sodiscoveryandroutingcannotdisagree. | | | | | | Ap- | | | | | | | |
| | | | | | | | nized fields | when | they | echo the | conversation, | | which would |
pendixAgivesanabridgedregistryexcerpt.
| | | | | | | | breakmulti-turntooluse. | | | Ontheresponse,thecodecthere- | | | |
| --------- | --- | ---------- | --- | --------------- | --- | ----------- | ----------------------- | --- | --- | ---------------------------- | --- | --- | --- |
| Admission | and | discovery. | | Model discovery | | is rendered | | | | | | | |
forereversiblyencodesthesignatureintothetool-calliden-
fromtheregistry,anditsvisibilityisconfigurablepermodel.
tifier, whichclientsmustechoverbatimtocorrelatetoolre-
Thecallerisauthenticatedbeforethemodelislookedup,and
| | | | | | | | sults. Onthenextrequest, | | | itdecodestheidentifier, | | | restores |
| --- | --- | --- | --- | --- | --- | --- | ------------------------ | --- | --- | ----------------------- | --- | --- | -------- |
unknownandpolicy-deniedmodelsreturnthesameresponse
| | | | | | | | the original | ID | and reinstates | | the signature | in | the extension |
| --- | --- | --- | --- | --- | --- | --- | ------------ | --- | -------------- | --- | ------------- | --- | ------------- |
(HTTP404,“modelnotfoundornotavailable”),sotheAPI
| | | | | | | | field before | forwarding | | upstream. | The | mechanism | is state- |
| -------- | ------ | ------- | -------- | ----------------- | --- | ----- | ------------ | ---------- | --- | --------- | --- | --------- | --------- |
| does not | reveal | whether | a hidden | or limited-access | | model | | | | | | | |
lessandappliesonlytosignaturesattachedtotoolcalls;the
| exists. | The specific | reason | is | recorded | server-side, | in logs | | | | | | | |
| ------- | ------------ | ------ | --- | -------- | ------------ | ------- | --- | --- | --- | --- | --- | --- | --- |
AnthropicMessagesandOpenAIResponsescodecsusethe
andauditreasoncodes,foroperators.
sameidentifierencoding.
| From file | to database. | | The | registry | began as | a version- | | | | | | | |
| --------- | ------------ | --- | --- | -------- | -------- | ---------- | ---------- | ----------------------------------------- | --- | --- | --- | --- | --- |
| | | | | | | | Streaming. | Server-senteventstreamsareconvertedincre- | | | | | |
controlledfileandnowhasadurablestorebehindtheman-
| | | | | | | | mentally. | AdecodermapswirechunkstoIRstreamevents | | | | | |
| ---------- | ------------ | -------- | ----------------- | -------------- | --------- | --- | ------------- | -------------------------------------- | ------------- | ------- | ------ | ---------- | ------------- |
| agement | API (section | | 10.1). A | leader-elected | projector | re- | | | | | | | |
| | | | | | | | (message | start, | content-block | | start, | text and | tool-argument |
| builds the | served | registry | deterministically | | whenever | the | | | | | | | |
| | | | | | | | deltas, block | | stop, usage, | message | | stop), and | an encoder |
store’sepochchangesandpublishesitthroughthesamebun-
| | | | | | | | mapsthoseeventstothetarget’seventgrammar. | | | | | | Per-stream |
| ------------------------------ | -------- | -------------- | ------- | --------------------- | ---------- | -------- | ----------------------------------------- | ---------- | ----- | ------------- | ------- | ------------- | ------------ |
| dlechannelaspolicy(section10). | | | | Thedataplanevalidates | | | | | | | | | |
| | | | | | | | decode | and encode | state | carries | partial | bytes | across chunk |
| the projected | registry | | when it | loads it; | an invalid | or miss- | | | | | | | |
| | | | | | | | boundaries, | maps | the | wire stream’s | | content-block | indices |
| ing bundle | fails | configuration. | | Registry | and policy | epochs | | | | | | | |
ontotheIR’sindexedblocks,andbuffersnothingbeyondan
| are independent, | | so a | catalog change | never | requires | rewrit- | | | | | | | |
| ---------------- | --- | ---- | -------------- | ----- | -------- | ------- | ---------- | ------ | --------- | --- | ---------- | ---- | ------------ |
| | | | | | | | incomplete | event. | Reasoning | | signatures | that | some vendors |
ingpolicyrules,andviceversa;registryandaccessdataare
requiretobeechoedonthenextturnarecarriedthroughthe
publishedinthesamebundle.
| | | | | | | | IR, so multi-turn | | tool | use survives | a | format | change. Token |
| --- | --- | --- | --- | --- | --- | --- | ----------------- | --- | ---- | ------------ | --- | ------ | ------------- |
usageistappedfromthestreamandreportedintrailerswith-
| 7 In-Proxy | | Translation | | | | | | | | | | | |
| -------------- | --- | --------------------------- | --- | --- | --- | --- | ------------ | ------ | ----------- | ------ | -------- | ------- | ------------ |
| | | | | | | | out altering | the | passthrough | | bytes. | Because | Proxy-Wasm |
| | | | | | | | forbids | header | writes | during | the body | phase, | all metadata |
| AcanonicalIR:N | | codecs,notN(N−1)converters. | | | | Ev- | | | | | | | |
producedwhilestreamingleavesthefilterastrailers.
erywireformatdecodesinto,andencodesoutof,onecanon-
| | | | | | | | Schema | drift | is a signal, | not | silent | loss. | Vendors evolve |
| --- | --- | --- | --- | --- | --- | --- | ------ | ----- | ------------ | --- | ------ | ----- | -------------- |
icalintermediaterepresentation(IR).Anany-to-anyconver-
sionisdecode→IR→encode. Eachcodectranslatesonly their APIs continuously. Request fields the codecs cannot
| | | | | | | | classifyarecountedandsurfacedasatrailerandametric. | | | | | | In |
| ------- | ------ | --- | -------- | ------- | ------- | --------- | -------------------------------------------------- | --- | --- | --- | --- | --- | --- |
| between | the IR | and | one wire | format, | with no | knowledge | | | | | | | |
addition,acoveragetestfailsthebuildwhenaknownfieldis
| of other | formats, | so adding | a format | costs | one codec | rather | | | | | | | |
| -------- | -------- | --------- | -------- | ----- | --------- | ------ | --- | --- | --- | --- | --- | --- | --- |
unclassified,whichforcesahumandecisioninsteadofsilent
| thanoneconverterperexistingformat. | | | | | Thesystemhasfive | | | | | | | | |
| ---------------------------------- | --- | --- | --- | --- | ---------------- | --- | --- | --- | --- | --- | --- | --- | --- |
dataloss.
general-purposecodecs,plusrawadaptersforasmallnum-
berofinternalformats. Gateway-side prompt caching. Coding agents resend
6
long, stableprefixes(systemprompts, tooldefinitions, con- Request Detect Applypolicy Model
versation history) on every turn. The filter injects cache maycontainpersonaldata rules+ block·redact· provider
| | | | | | | | | | internalmodel | | pseudonymize | | |
| ----------- | --- | -------- | ---------------- | --- | ---------- | ------ | --- | --- | ------------- | --- | ------------ | --- | --- |
| breakpoints | on | the last | tool definition, | | the system | prompt | | | | | | | |
andthemostrecentcontentblock,withinthevendor’slimit
| | | | | | | | Response | | | Restoreoriginalvalues | | | |
| ------- | ----------- | --- | ----------- | -------- | --- | ----------- | -------- | --- | --- | --------------------- | --- | --- | --- |
| of four | breakpoints | [3] | and without | removing | | breakpoints | | | | | | | |
theclientalreadyset,anditrequestsanextended(one-hour)
Figure3:Data-loss-preventionflow.Detectedsensitivecontentisblocked,
| cache lifetime | | for all | of them. | This shaping | | applies on the | | | | | | | |
| -------------- | --- | ------- | -------- | ------------ | --- | -------------- | --- | --- | --- | --- | --- | --- | --- |
redactedorreplacedwithformat-preservingsurrogatesinsidetheenterprise
Anthropic-on-Vertexupstreampath. Forprovidersthatsup- boundarybeforetheprovidercall; surrogatesaremappedbackinthere-
| port it, | the control | service | can | also | add a | stable, hashed | sponse. | | | | | | |
| ---------------- | ----------- | ----------- | ---- | ---- | -------- | -------------- | --------------- | --- | ---------------------------------- | --- | --- | --- | --- |
| session-affinity | | key derived | from | the | caller’s | identity (and | | | | | | | |
| | | | | | | | (OIDC)[22]flow. | | Themintserviceacceptsonlyaverified | | | | |
theclient’ssessionwhereavailable),enabledperprovideror
id_tokenwhoseauthorizedparty(or,whenabsent,audi-
foraconfigurableshareoftraffic,sothatrepeatedturnsland
ence)resolvestoaregistered,activeplatform.Theidentity
| onwarmcaches. | | Clientsgetthesebenefitswithoutchanging | | | | | | | | | | | |
| ------------- | --- | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
providerissuessuchtokensonlythroughthatclient’sflow.
anycode;theeffectisreportedinsection13.
| | | | | | | | • The user | is the | subject; | the | platform | is an | attribu- |
| ----------- | --- | ----------- | --- | --- | ---- | --- | -------------- | ------ | ---------------------------------- | --- | -------- | ----- | -------- |
| | | | | | | | tiondimension. | | Theplatformcannotchoosethesubject. | | | | |
| 8 Identity, | | Credentials | | and | Data | | | | | | | | |
Spendischargedtotheuser’sbudget,andtheplatformap-
Protection
pearsoneveryusagerecord.
| | | | | | | | • Offline | verification. | | Tokens | are | signed, | with keys |
| ------- | ------------ | --- | ----- | ------ | ------ | ----------- | --------- | ------------- | ------ | ------ | --- | ---------- | --------- |
| Inbound | credentials. | | Every | caller | of the | unified API | | | | | | | |
| | | | | | | | published | through | a JSON | Web | Key | Set (JWKS) | end- |
presentsaninternalcredential,neveravendorkey. Develop- point[10]. Thecontrolserviceverifiesthemlocally,with
ers use a personal access token tied to their corporate iden- noper-requestcalltothemintservice.
| tity. Business | applicationspresent | | | an | application | credential | | | | | | | |
| -------------- | ------------------- | --- | --- | --- | ----------- | ---------- | ------------------ | --- | -------------------------------- | --- | --- | --- | --- |
| | | | | | | | • Boundedlifetime. | | Tokenlifetimeiscappedperplatform | | | | |
boundtoaregisteredapplication,sothatusageisattributed andneverexceedstheexpiryofthepresentedidentityto-
| to that | application | rather | than | to an individual. | | Platforms | | | | | | | |
| ------- | ----------- | ------ | ---- | ----------------- | --- | --------- | ------- | ---------- | ------ | ------ | ------ | ------ | -------- |
| | | | | | | | ken, so | re-minting | cannot | extend | access | beyond | what the |
thatactonbehalfofuserspresenttokensfromthemintser- identityproviderissued.
vicedescribedbelow. Personalandplatform-mintedtokens • Tags are attribution, never authority. Tag keys are
resolvetoaperson,whomthepolicylayerevaluatesagainst
allow-listedperplatform,size-boundedandrejectedwhen
theorganizationalhierarchy; applicationcredentialsresolve unknown. They never expand routing, authorization or
totheregisteredapplication.
quota.
Outbound: brokered, keylesscredentials. Clientsofthe • Authentication tokens do not persist. The id_token
| | | | | | | | and the | minted | token are | never | logged | or stored. | Every |
| ------- | --------- | ---- | -------- | ---- | ----------- | -------- | ------- | ------ | --------- | ----- | ------ | ---------- | ----- |
| unified | API never | hold | a vendor | key, | so a leaked | applica- | | | | | | | |
tionconfigurationdoesnotexposeavendorcredential. The decision,includingrejections,isauditedbyreasoncode.
Mintedtokenscanalsocarryaworkflowbudget,boundedat
registrydeclaresacredentialkindperproviderandthegate-
way resolves it per request: per-user upstream credentials mint time and signed, that caps the total spend of a work-
brokered by the gateway, OAuth2 client-credentials grants flowacrossallofitsusers. Itisenforcedinadditiontouser
budgetsandcanneverbypassthem.
fordirectory-backedproviders,thegateway’sownworkload
identityforcloud-hostedmodels,and,forapplicationtraffic,
8.1 Datalossprevention
| ashort-livedcredentialscopedtothecallingapplication. | | | | | | Se- | | | | | | | |
| ---------------------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
cretsarefetchedatruntimefromamanagedsecretstoreus- Prompts in a payments company can contain personal and
ingtheplatform’sownidentity;secretvaluesarenotplaced payment data that should not reach an external model
in code or static configuration, and request logging redacts provider. Forusecaseswhosedataclassificationrequiresit,
credentials. the gateway therefore applies a data-loss-prevention (DLP)
SSO-attested platform identity. Platforms that call the layerontherequestpath(fig.3).
gatewayonbehalfofusers, suchasagentharnessesandin- Detectioninsidetheenterpriseboundary. Detectingsen-
ternaldeveloperplatforms, needa trustworthywaytoiden- sitive data in free text is harder than matching patterns: the
tify themselves and label workflows. We introduced a mint same date or number can be harmless in one sentence and
servicethatconverts“ausersignedintoplatformPthrough sensitiveinthenext.Detectionthereforecombinesdetermin-
corporateSSO”intoasigned,time-boundedJSONWebTo- isticpatternruleswithaninternallyhosted,purpose-trained
ken (JWT) [11] naming the user, the platform and optional detectionmodelthatusesthesurroundingcontext. Bothrun
attributiontags(e.g., workflowandrun). Itsdesignchoices inside the enterprise boundary, so content is inspected be-
| are: | | | | | | | foreanyvendorcallismade. | | | Detectionisprobabilistic,and | | | |
| ---- | --- | --- | --- | --- | --- | --- | ------------------------ | --- | --- | ---------------------------- | --- | --- | --- |
• No shared platform secret. A platform is identified by accuracyvariesbyentitytype. DLPthereforereducesexpo-
the OAuth client identifier bound to its OpenID Connect sure rather than guaranteeing that no sensitive value is ever
7
sent,anditcomplements,ratherthanreplaces,use-casedata Layer Setby Effect
classificationandprovidercontractualcontrols.
| | | | | | | Catalogadmission | | platform(registry) | | unknownmodel→404 | | |
| --- | --- | --- | --- | --- | --- | ---------------- | --- | ------------------ | --- | ---------------- | --- | --- |
Policy-drivenactions. Eachusecaseisconfiguredwithits Modelrestriction platformadmin leader allow/deny lists and
exemptions→404(sameas
| own DLP | policy, because | data | classification, | | security zone | | | | | | | |
| ------- | --------------- | ---- | --------------- | --- | ------------- | --- | --- | --- | --- | --- | --- | --- |
unknown)
| and tolerance | for false | positives | differ | between | use cases. | | | | | | | |
| ------------- | --------- | --------- | ------ | ------- | ---------- | ----------------- | --- | ------------- | --- | ----------------------- | --- | --- |
| | | | | | | BUallow/blocklist | | platformadmin | | amodelisopento,orclosed | | |
From most to least restrictive, a policy can block a request to,namedbusinessunits
| | | | | | | BUmodeltoggle | | BUadmin | | switchamodeloffforone’s | | |
| ------------- | --------- | ------------ | --- | --------- | -------- | ------------- | --- | ------- | --- | ----------------------- | ------ | ---------- |
| that contains | sensitive | data; redact | the | sensitive | spans by | | | | | | | |
| | | | | | | | | | | own BU; | cannot | override a |
maskingorremovingthem;orpseudonymizethem,replacing
platformlock
eachvaluewitharealisticsurrogateofthesameformatand Limitedaccess platformadmin bounded named roster with
restoring the original values in the response. For these use its own spend controls and
expiry
| casesthegatewayalsohandlestheresponsebody: | | | | | theorig- | | | | | | | |
| ------------------------------------------ | ----------------- | --- | ------- | -------- | ------------ | ----------- | --- | ------------- | --- | ------------ | -------------- | --- |
| | | | | | | Pilotcohort | | platformadmin | | date-bounded | allow/restrict | |
| inal values | that were swapped | | out are | held for | the duration | | | | | | | |
listspercohort
oftherequestandsubstitutedbackfortheirsurrogatesbefore
theresponseisreturnedtotheclient.Redactionissimplebut Table4: Access-controllayers. Eachlayernarrowstheaccessgrantedby
candistortthetask(amodelaskedtoapplyanage-basedrule theoneaboveit,exceptauthoritativepilotcohorts(section10.2);spendcaps
cannotreasonaboutamaskeddate). Pseudonymizationpre- areseparateandcanberaisedwithinplatformlocks(section9.2).
| serves the | format of each | detected | value | while | the provider | | | | | | | |
| ---------- | -------------- | -------- | ----- | ----- | ------------ | --- | --- | --- | --- | --- | --- | --- |
rectorythroughacache,withtheorganizationsnapshotdis-
neverseesthedetectedoriginals,whichmakesitwellsuited
| | | | | | | tributedinthepolicybundleasafallback(section10). | | | | | | Be- |
| --- | --- | --- | --- | --- | --- | ------------------------------------------------ | --- | --- | --- | --- | --- | --- |
toconversationalusecases.
| | | | | | | cause rules | name | leaders | rather | than team | lists, a | reorgani- |
| --- | --- | --- | --- | --- | --- | ----------- | ---- | ------- | ------ | --------- | -------- | --------- |
Operational properties. DLP is integrated into the gate- zationthatmovesateamautomaticallymovesitsaccessand
| way API | and is enforceable | per | use | case: an | application | | | | | | | |
| ------- | ------------------ | --- | --- | -------- | ----------- | --- | --- | --- | --- | --- | --- | --- |
budgets,withnopolicyedits.
| adopts it | through per-use-case | | configuration | and | a request- | | | | | | | |
| ----------------- | -------------------- | ------ | ------------- | ------------- | ---------- | ---------- | --- | ------------------------- | --- | --- | --- | --- |
| | | | | | | 9.2 Spend: | | scopesareviews,notstorage | | | | |
| level enforcement | flag | rather | than an | extra service | call or | | | | | | | |
SDKchange. Inspectionfailsclosed:ifacheckcannotcom- LetM bethesetofcanonicalmodelidentifiersandτ:M
→
2T
plete,therequestisnotsent. DLPeventsareloggedforse- assigneachmodelasetoftagsfromT,thesetofregistry
curitymonitoringandaudit. Togetherwithamediatedpath, tags, along several axes: everything, vendor, model family,
attribution and consistent usage records, these are techni- license class (proprietary or open-weight), or any operator-
calcontrolsthatcancontributetotheorganization’sbroader defined axis. A scope s is a tag or a single model, and re-
complianceprograms; thegatewaydoesnotbyitselfestab- M ={m:s∈τ(m)}∪({s}∩M).
| | | | | | | solves to | s | | | | For | budget- |
| --- | --- | --- | --- | --- | --- | --------- | --- | --- | --- | --- | --- | ------- |
lishcompliancewithanyregulatoryframework. ing, spend for user budgets is accumulated only as coun-
tersc(u,m,ω)inmicro-dollars,keyedbyuseru,immutable
| 9 The | Policy Model | | | | | | | | | | | |
| ----- | ------------ | --- | --- | --- | --- | ---------------------------------- | --- | --- | --- | --- | --------------- | --- |
| | | | | | | modelidentifiermandwindowinstanceω | | | | | (daily,weeklyor | |
monthly,onasinglefixedcalendartimezoneforallregions).
whomay
Governanceanswersthreeindependentquestions: Here u ranges over people; application traffic is attributed
usewhichmodel,howmuchmaybespent,andhowfast.The
byapplicationidentifier(section11),andworkflowbudgets
| policies | in this section | govern | person-attributed | | traffic: de- | | | | | | | |
| -------- | --------------- | ------ | ----------------- | --- | ------------ | --- | --- | --- | --- | --- | --- | --- |
(section8)areaccumulatedincountersoftheirown.
| velopers, | and users acting | through | platforms. | | Application | | | | | | | |
| --------- | ---------------- | ------- | ---------- | --- | ----------- | -------- | ---- | -------- | --- | ----------- | ------- | --- |
| | | | | | | A budget | rule | consists | of | one or more | entries | e = |
trafficisgovernedthroughitsapplicationregistrationandat-
| | | | | | | (P,s ,w | ,θ ,A | ), eachwithapopulationpredicateP | | | | (user |
| --- | --- | --- | --- | --- | --- | ------- | ----- | -------------------------------- | --- | --- | --- | ----- |
tributed by application identifier in the usage record (sec- e e e e e e
tiers,organizationleaders,businessunit,cohortandexemp-
tion11).
| | | | | | | tions), a | scope, | a window, | a threshold | and | a set of | actions. |
| --- | --- | --- | --- | --- | --- | --------- | ------ | --------- | ----------- | --- | -------- | -------- |
Rulescarryaprecedenceweight,andentriesareorderedby
9.1 Modelrestrictionsandaccesslayers
| | | | | | | it. Theutilizationofentryeforuseruattimet | | | | | is | |
| ------------------------ | --- | -------------------------- | --- | --- | --- | ----------------------------------------- | --- | --- | --- | --- | --- | --- |
| Accessislayered(table4). | | Eachaccesslayercanonlynar- | | | | | | | | | | |
1
row what the layers above it allow (an authoritative pilot’s ∑ (cid:0) (cid:1)
| | | | | | | | ρ e | (u,t)= | c | u,m,ω we | (t) . | |
| -------------------------------------------------------- | --- | --- | --- | --- | --- | --- | --- | ------ | --- | -------- | ----- | --- |
| cohortlistsdecideaccessforitsenrolledusers,section10.2), | | | | | | | | θ | | | | |
em∈M
| and each | is evaluated | before | any vendor | credential | is is- | | | | se | | | |
| -------- | ------------ | ------ | ---------- | ---------- | ------ | --- | --- | --- | --- | --- | --- | --- |
m∗,
sued. Spend caps work differently: an approved business- For a request by u for model the request is blocked
unit override can raise a platform cap, but only within the by the first entry, in precedence order, that has a block ac-
platform’slocks(sections9.2and10.2). Modelrestrictions tion,satisfiesP(u)∧ρ (u,t)≥1,coverstherequestedmodel
e e
∈M
name organization leaders rather than teams: a model may (m∗ ), and is not superseded. Otherwise it is paused
se
carryaleaderdenylist,anoptionalleaderallowlist,andex- by the first matching entry with an active pause that cov-
ersm∗.
plicitper-person exemptions. Acallerbelongs toaleader’s Scopingblockstotherequestedmodelmeansacap
organizationwhenthatleaderappearsinthecaller’supward ononemodelfamilystopsonlythatfamily: auserwhoex-
reporting chain. The chain is resolved from the people di- hausts a frontier-model budget can keep working on other
8
models. A platform entry is superseded for a user when an package gateway.spend
| approvedbusiness-unitoverrideforthatuser’sBUcoversthe | | | | | | | | import rego.v1 | | | | | | |
| ----------------------------------------------------- | --- | --- | --- | --- | --- | --- | --- | -------------- | --- | --- | --- | --- | --- | --- |
import data.gateway.util
| same window | and | scope | with | a higher | threshold, | | so an ap- | | | | | | | |
| ----------- | --- | ----- | ---- | -------- | ---------- | --- | --------- | --- | --- | --- | --- | --- | --- | --- |
proved BU increase outranks the platform baseline for that # entries that apply to this caller (tier, org, BU,
| | | | | | | | | # cohort, | exemptions), | with | utilization | in percent | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --------- | ------------ | ---- | ----------- | ---------- | --- | --- |
BUonly. Separately,thegoverninglimitreportedtotheuser
| | | | | | | | | applicable | contains | entry if | { | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | ---------- | ----------------------- | -------- | --- | --- | --- | --- |
| | | | | | | | | some e | in data.gateway.entries | | | | | |
istheblockorpauseentrywiththehighestutilization.
| | | | | | | | | util.population_matches(e, | | | input) | | | |
| -------- | ------------ | ------------- | --- | --------- | ---- | ---------- | ----- | -------------------------- | ------------------------------------- | --- | ------- | ----- | ------------ | --- |
| Actions | compose. | Block | is | stateless | and | recomputed | per | | | | | | | |
| | | | | | | | | pct := | util.pct(input.spent[e.scope_window], | | | | e.threshold) | |
| | | | | | | | | entry := | object.union(e, | | {"pct": | pct}) | | |
| request, | so it clears | automatically | | | when | the window | rolls | | | | | | | |
}
over.Pauseblocksforafixeddurationandfiresatmostonce
perruleandwindow,soauserwhoreturnshourslaterisnot blocking := util.first_by_precedence([e |
| | | | | | | | | some e | in applicable | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | ------ | ------------- | --- | --- | --- | --- | --- |
penalized again. Message adds a live status line, and email some a in e.actions
notifiestheuseror,asadeferreddigest,theleadershipchain a.type == "block"
| | | | | | | | | e.pct >= | 100 | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | -------- | --- | --- | --- | --- | --- | --- |
(section10.3). Aseparateno-promoteactioncanbarapop- util.covers(e.scope, input.model)
not util.superseded_by_bu_override(e)
ulationfromself-servicetierpromotion(section9.3).
])
Thisformulationhasapropertythatmattersoperationally.
| | | | | | | | | decision | := {"block": | true, | "limit": | blocking.id} | if { | |
| ------- | -------- | -------- | --- | ---------- | ---- | ---- | -------- | -------- | ------------ | ----- | -------- | ------------ | ---- | --- |
| Because | tags are | resolved | at | evaluation | time | over | counters | | | | | | | |
input.enforce
| keyedbyimmutablemodelidentifiers,anewlycreatedtag | | | | | | | | blocking | | | | | | |
| ------------------------------------------------- | ----------- | --- | ------- | ----- | -------- | ------- | --- | --------- | --------- | ------ | --- | --- | --- | --- |
| | | | | | | | | } else := | {"block": | false} | | | | |
| or rule | immediately | | governs | spend | recorded | earlier | in | | | | | | | |
the current window, before the rule existed. No backfill, Listing 1: Simplified, illustrative spend decision in Rego (not the
key rewrite or migration is needed. A business-unit policy productionpolicy).Spendsumsaresuppliedbythegateway;thepolicyonly
decides. Inshadowmodenothingblocks,bututilizationisstillcomputed
| such as | “cap each | person | in | this BU | at a | monthly | amount | | | | | | | |
| ------- | --------- | ------ | --- | ------- | ---- | ------- | ------ | --- | --- | --- | --- | --- | --- | --- |
andreported.
forOpus-classmodels”(avendor’slargestmodelfamily)is
asingledatachangethattakeseffectoverthemonthtodate.
Business-unitpoliciesareentrieswhosepopulationpredicate (scope, window), pause state, window boundaries and the
selects that unit; their caps apply per person (section 10). enforcementswitch. Evaluationreturnsadecision(blockor
Retroactive scope is not retroactive repricing: counters are allow, thelimitingentry, aretry-afterhintandper-limituti-
already denominated in micro-dollars at the price in force lization). The gateway then performs any side effects: up-
wheneachrequestcompleted,soalaterpricechangeaffects datingcounters,recordingpausesandone-shotnotifications,
onlysubsequentrequests.Thesamepropertyletsadraftrule andqueuingemails. Listing1givesasimplifiedillustration.
| be dry-run | against | current | usage | before | it | is enabled | (sec- | | | | | | | |
| ---------- | ------- | ------- | ----- | ------ | --- | ---------- | ----- | ---------- | --- | --------- | ---- | ----------------- | --- | ------- |
| | | | | | | | | The policy | is | organized | into | isolated modules: | | access, |
tion10). AppendixBshowstherulelanguage. spend,ratelimiting,alerts,pilots,validation,sharedutilities
andread-onlyadministrativequeries.Thepilotmoduleisde-
| Rate limits | and | workflow | budgets. | | Per-minute | | rate lim- | | | | | | | |
| ----------- | --- | -------- | -------- | --- | ---------- | --- | --------- | --- | --- | --- | --- | --- | --- | --- |
its are available to protect shared upstream quotas. Work- liberatelyisolatedfromplatformandBUpolicydocuments,
| | | | | | | | | andthepolicytesttargetchecksthatseparation. | | | | | Onthere- | |
| --- | --- | --- | --- | --- | --- | --- | --- | ------------------------------------------- | --- | --- | --- | --- | -------- | --- |
flowbudgets(section8)applyinadditiontouserrules,and
whicheverlimittripsfirstblocks. questpath,anenrolledpilotdecisionoverridestheplatform
| | | | | | | | | and BU decision | | only when | pilot | enforcement | is authorita- | |
| ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --------------- | --- | --------- | ----- | ----------- | ------------- | --- |
| 9.3 Usertiersandself-servicepromotion | | | | | | | | tive. | | | | | | |
Users belong to one of three tiers with progressively larger Keeping decisions pure gave us four practical benefits.
| | | | | | | | | Policies are | unit-testable, | | and the | policy suite | requires | cov- |
| ----------- | ------- | ---- | ---------- | --- | ----------- | ------ | ---- | ------------ | -------------- | --- | ------- | ------------ | -------- | ---- |
| allowances, | derived | from | enterprise | | entitlement | roles. | Pro- | | | | | | | |
motion is self-service through an internal access workflow. erage of every production module. A “what-if” admission
Promotion is subject to usage-based eligibility, and promo- querycanpreviewaproposedchangewithouttouchinglive
tionsexpireautomatically. Anybudgetentrycanalsocarry state. The policy and organization data are distributed as a
ano-promotegate,eitherunconditionallyforapopulationor versioned bundle (section 10), so every data-plane replica
| | | | | | | | | evaluates | the same | revision. | And | the migration | itself | was |
| ---------- | --- | ---------- | ------ | --- | ----- | ---------- | ----- | --------- | -------- | --------- | --- | ------------- | ------ | --- |
| only while | the | requesting | user’s | own | spend | in a scope | is at | | | | | | | |
oraboveathreshold,soleadershipcanpausepromotionsfor safe:thepreviousGoengineandtheOPAengineransideby
anorganizationwithouteditinguserrecords. sidebehindaselectorwhileaparitysuitecheckedidentical
decisionsacrossallcombinationsofengineandenforcement
9.4 Apuredecisionfunction
mode,togetherwithhot-reloadtestsofupdatedrules.
| We moved | policy | evaluation | | from | imperative | | code to | | | | | | | |
| -------- | ------ | ---------- | --- | ---- | ---------- | --- | ------- | --- | --- | --- | --- | --- | --- | --- |
OPA [17] and its policy language, Rego. The dividing line 10 The Governance Control Plane
| is strict: | the gateway | | gathers | state | and performs | | effects; | | | | | | | |
| ---------- | ----------- | --- | ------- | ----- | ------------ | --- | -------- | --- | --- | --- | --- | --- | --- | --- |
Rego decides. Before evaluation, the control service as- Apolicyengineisusefulonlyiftherightpeoplecanchange
sembles an input document containing the caller, canoni- policy safely. The control plane turns models, prices and
cal model, tier, organization chain, the summed spend per policy from engineering artifacts into an operated product
9
| | | | | Validate | | | Approve | Schedule | | Persist+audit | | | | |
| --- | ------ | --- | --- | -------- | --- | --------------- | ------- | -------- | --- | ------------- | --- | --- | ------- | --- |
| | Author | | | | | BUchangesqueued | | | | | | | Project | |
platformorBUadmin schema,conflicts, activateatlocal durablestore,epoch, leader-elected,
orauto-approved;
(SSO-derivedrole) locks,dry-run noself-approval midnight,ornow activitylog deterministicbundle
informnextchange
| | | Feedback | | | | | Rolloutmodes | | | Enforce | | | Distribute | |
| --- | --- | -------- | --- | --- | --- | --- | ------------ | --- | --- | ------- | --- | --- | ---------- | --- |
utilization%,thresholdemails,statuslines, trackingalwayson→shadow(evaluate, everyrequest; versioned;conditional
leadershipdigests,savedvs.enforcedrevision log,neverblock)→authoritative(enforce) last-goodonfailure polling;hotreload
Figure4: Policylifecycle. Intentisauthoredandvalidatedagainstthesamerulesthedataplaneenforces,approvedwhenitoriginatesfromabusinessunit,
persistedwithanauditrecord,projectedintoaversionedbundle,andenforcedoneveryrequest.Newcontrolsarerolledoutinshadowmodebeforetheyblock.
1Createmodelbrand,family,formats,aliases,tags→staged accepted,andcapsmustbestrictlyorderedacrosswindows
| | | | | | | | | (a monthly | cap | exceeds the | weekly | cap, which | exceeds | the |
| --- | ------------------------------------------------------------- | --- | --- | --- | --- | --- | --- | ---------- | ------------------------------------------- | ----------- | ------ | ---------- | ------- | --- |
| | 2Adddeploymentsprovider,region,priority,pricepermilliontokens | | | | | | | dailycap). | | | | | | |
| | | | | | | | | BUpolicies | letbusiness-unitadministratorssetper-person | | | | | |
3Validate&publishtopology,capabilities,pricing;projectorpublishes and per-model caps and model toggles for their own orga-
| | | | | | | | | nization, | within | the platform’s | locks: | an | unlocked | BU cap |
| --- | --- | --- | --- | --- | --- | --- | --- | --------- | ------ | -------------- | ------ | --- | -------- | ------ |
4Limitedaccess/pilotboundedroster,owncaps,expiry;shadowfirst takesprecedenceovertheplatformbaseline,arequestabove
alockedmaximumisrejected(andanystoredcapisclamped
5Graduategenerallyavailable;tag-basedpoliciesapply
| | | | | | | | | to that maximum | | when policies | are | applied), | and | an exact |
| --- | --- | --- | --- | --- | --- | --- | --- | --------------------- | --- | ----------------------------- | --- | --------- | --- | -------- |
| | | | | | | | | lockrejectsanychange. | | BUmodeltogglestakeeffectwith- | | | | |
Figure5:Modelonboardingthroughthecontrolplane:eachstepisavali-
| | | | | | | | | out approval | but | cannot re-enable | | a model | the platform | has |
| --- | --- | --- | --- | --- | --- | --- | --- | ------------ | --- | ---------------- | --- | ------- | ------------ | --- |
dated,auditeddatachange;nocodereleaseisinvolved.
locked. Controlscanselectmodelsexplicitlyorbytag,and
tagselectionsfollowtheregistry.
(fig.4).
| | | | | | | | | Approvalsandscheduling. | | | BUcapchangesareeithersub- | | | |
| -------------- | --- | --- | ------------------------------------- | --- | --- | --- | --- | ----------------------- | --- | --- | ------------------------- | --- | --- | --- |
| Rolesandtrust. | | | Administratorsauthenticatethroughcor- | | | | | | | | | | | |
porate single sign-on, and the gateway maps each adminis- mitted to an approval queue, where they do not affect en-
trator onto one of two canonical roles. Platform adminis- forcement until approved, or auto-approved where the plat-
trators act organization-wide. BU administrators act only formallowsit. Decisionsarerestrictedtoplatformadminis-
withinthebusinessunitstheyhavebeengranted. trators,self-approvalisforbidden,arejectionmustincludea
comment,andanapprovalcanbeboundtotherevieweddiff
10.1 Modelonboardinglifecycle
| | | | | | | | | so that edits | arriving | in between | are | detected. | Changes | can |
| --- | --- | --- | --- | --- | --- | --- | --- | ------------- | -------- | ---------- | --- | --------- | ------- | --- |
Onboardingamodelisasequenceofvalidateddatachanges bescheduledtoactivateatmidnightinagiventimezone. A
through the management API (fig. 5). A platform adminis- leader-gated scheduler applies them, expires stale requests,
trator creates the model (brand, API family, entry formats, and re-projects whenever the store’s epoch is ahead of the
aliases, tags, capabilities) and then its deployments, each servedbundle,sothedurablerecordandtheenforcedbundle
| withprovider,region,priorityandprice. | | | | | | Everywriteisval- | | converge. | | | | | | |
| ------------------------------------- | ------- | --- | -------- | --------- | ---------- | ---------------- | ----------- | --------- | --- | --- | --- | --- | --- | --- |
| idated | against | the | registry | topology, | capability | | and pricing | | | | | | | |
Pilots. Newmodelsandnewbudgetregimesareintroduced
rules;updatesanddeletesrequireanentity-tagprecondition
| | | | | | | | | through | pilots: | named, date-bounded | | programs | | of a small |
| --- | --- | --- | --- | --- | --- | --- | --- | ------- | ------- | ------------------- | --- | -------- | --- | ---------- |
(missingreturns428,stalereturns409),andanidempotency number of cohorts. Each cohort has its own cost controls,
| keymakesretriessafe. | | | | Untilithasanenableddeployment, | | | | | | | | | | |
| -------------------- | --- | --- | --- | ------------------------------ | --- | --- | --- | ------------ | ------ | -------------- | --- | ------ | --------------- | --- |
| | | | | | | | | model-access | lists, | BU assignment, | | alerts | and exceptions. | |
amodelisstaged andnotserved. Anewmodelcanfirstbe Cohorts that share a leader and window but have distinct
| released | as | limited | access | to a bounded | | named | roster, with | | | | | | | |
| -------- | --- | ------- | ------ | ------------ | --- | ----- | ------------ | ----------------------- | --- | --- | -------------------------- | --- | --- | --- |
| | | | | | | | | amountsformatierladder. | | | Anenrolleduserisgovernedby | | | |
itsownspendwindows,amountsandexpiry,orthroughapi-
thelowesttierwhoseexceptionsdonotcoverthem,soanex-
lot. Itis thengraduated, whichclearstherosterandmakes ception promotes a user up the ladder rather than removing
| the | model | generally | available. | Because | | policies | can select | | | | | | | |
| --- | ----- | --------- | ---------- | ------- | --- | -------- | ---------- | -------- | ---------- | ------- | ----- | --------- | ------- | ------- |
| | | | | | | | | the cap. | Validation | returns | every | violation | at once | (double |
modelsbytag,anewlytaggedmodeliscoveredbyexisting enrollment, duplicate controls, duplicate tier amounts, un-
| restrictions | | and | budgets | within one | publication | | cycle, with | | | | | | | |
| ------------ | --- | --- | ------- | ---------- | ----------- | --- | ----------- | -------- | ----- | ---------------- | --- | ------ | --------- | ------ |
| | | | | | | | | known or | empty | tag selections), | and | writes | serialize | across |
nopolicyedit.Amodelcannotbedeletedwhilealivepolicy
| | | | | | | | | replicas | under a | database advisory | | lock. | Pilot enforcement | |
| --- | --- | --- | --- | --- | --- | --- | --- | -------- | ------- | ----------------- | --- | ----- | ----------------- | --- |
referencesit,andeverystepisrecordedintheactivitylog. has its own switch: off, shadow (evaluate and log only) or
| 10.2 | Platformandbusiness-unitpolicies | | | | | | | authoritative. | | | | | | |
| ---- | -------------------------------- | --- | --- | --- | --- | --- | --- | -------------- | --- | --- | --- | --- | --- | --- |
10.3 Usagenotificationsandreminders
| Platform | | policies | set | organization-wide | | controls: | per- | | | | | | | |
| -------- | --- | -------- | --- | ----------------- | --- | --------- | ---- | --- | --- | --- | --- | --- | --- | --- |
person runaway-cost limits, per-model caps, model-access Usersshouldlearnthattheyareapproachingalimitlongbe-
decisions, and locks that BU policies cannot exceed. Con- fore they hit it. Notification points are configurable at two
flict checks scan the complete active set before a change is levels. In the rule language, any entry can attach an email
10
or status-line action at its own threshold; the typical pat- Telemetry[18]. Labelsidentifythecanonicalmodel,vendor
ternisawarningemailat90%ofanallowanceandablock and entry path, and aliases collapse to canonical identifiers
plusemailat100%(appendixB).Inmanagedpolicies,each sothateachlogicalmodeliscountedonce.Per-callerdimen-
controlcarriesalertsatadministrator-chosenpercentages(1– sionsarelimitedtoasmallsetofgovernanceevents(suchas
100%),eachwithatleastonechannel(emailorin-toolmes- limit blocks); full per-request attribution lives in the usage
sage) and a template. Templates are managed through the record,whichkeepsmetriccardinalityboundedandidentity
API, are scoped to the platform or to a business unit, and datainanaccess-controlledstore.
| support | placeholders | such | as utilization | | percentage, | thresh- | | | | | | | |
| ------- | ------------ | ---- | -------------- | --- | ----------- | ------- | -------------------------------- | --- | --- | --- | --- | ---------------- | --- |
| | | | | | | | Theaccesslogasthesystemofrecord. | | | | | Weinitiallyemit- | |
old,resumetime,model,window,personandbusinessunit.
| | | | | | | | ted usage | events | from the | control service | | through | an asyn- |
| --- | --- | --- | --- | --- | --- | --- | --------- | ------ | -------- | --------------- | --- | ------- | -------- |
Deliveryisdeduplicatedperpolicyowner,windowinstance,
| | | | | | | | chronous | in-memory | queue. | That design | | lost data | silently: |
| --- | --- | --- | --- | --- | --- | --- | -------- | --------- | ------ | ----------- | --- | --------- | --------- |
threshold,templateandchannel,soauserreceivesoneemail
whenthequeuewasfull,afterretrieswereexhausted,during
| per crossing | rather | than | one per | request. | Status headers | on | | | | | | | |
| -------------- | ------ | ---- | --------- | -------- | -------------- | ---- | ---------------- | ----------------------------------- | ---- | -------- | ------------ | --- | ---------- |
| | | | | | | | pod shutdown, | and | when | a client | disconnected | | before the |
| every response | report | the | governing | limit’s | utilization, | sta- | | | | | | | |
| | | | | | | | trailersarrived. | WemovedthedurablerecordtoEnvoy’sac- | | | | | |
tus,scope,resettimeandanypause,andcodingagentsand
cesslog,whichEnvoywritesforeverystream,includingre-
| thedeveloperCLIrenderthemasalivestatusline. | | | | | | Leader- | | | | | | | |
| ------------------------------------------- | --- | --- | --- | --- | --- | ------- | --------------------------------------------------- | --- | --- | --- | --- | --- | --- |
| | | | | | | | sets, immediaterejectionsandcontrol-serviceoutages. | | | | | | The |
shipnotificationsaredeferredintoonedigestperleaderafter
controlservicecontributesattribution(user,application,tier,
| the window | boundary. | | The usage | API | reports utilization | as | | | | | | | |
| ---------- | --------- | --- | --------- | --- | ------------------- | --- | --- | --- | --- | --- | --- | --- | --- |
platform,workflow,harness,requestedandcanonicalmodel,
percentages.
entryandupstreamformats)andthepricedcostbreakdown
10.4 Rollout,distributionandaudit asmetadataassembledfromin-memorystateonly. Assem-
| | | | | | | | blingthismetadataaddsnoI/Otoanyext_procphase. | | | | | | The |
| ------ | ------ | ----- | -------- | --------- | ------- | -------- | ---------------------------------------------------- | --- | --- | --- | --- | --- | --- |
| Shadow | first. | Usage | tracking | is always | on, and | enforce- | | | | | | | |
| | | | | | | | Wasmfiltercontributesper-fieldtokencountsastrailers. | | | | | | In- |
ment is a separate switch, set globally and per pilot. New terrupted streams still produce a usage record. Blocked re-
controlsarethereforerolledoutinshadowmodeagainstreal
quests(4xximmediateresponsesforauthenticationfailures,
traffic,withutilizationvisibletousersandadministratorsand invalidrequests,accessdenialsandlimits)carrythesameat-
nothing blocked, until their effect is understood. Because tribution, which makes rejected traffic visible to abuse and
scopesareviewsovercurrent-windowcounters,adraftrule
| | | | | | | | chargeback | analysis. | The | earlier event | pipeline | | was run in |
| --- | --- | --- | --- | --- | --- | --- | ---------- | --------- | --- | ------------- | -------- | --- | ---------- |
canalsobedry-runagainstcurrentusagebeforeitisenabled. parallelduringmigrationsothetwocouldbereconciled.
| A rule enabled | directly | | while enforcement | | is on applies | to | | | | | | | |
| -------------- | -------- | --- | ----------------- | --- | ------------- | --- | --- | --- | --- | --- | --- | --- | --- |
thewindowtodateandcanblockatonce,whichiswhynew From log line to warehouse. Each record is one JSON
| | | | | | | | lineonstandardoutput. | | | Anode-levelloggingagentshipsit | | | |
| --- | --- | --- | --- | --- | --- | --- | --------------------- | --- | --- | ------------------------------ | --- | --- | --- |
controlsgothroughshadowmodeorapilotfirst.
toamanagedlogservice,fromwhichtherecordsarerouted
| Distribution. | The | durable | store | is the | system of | record. | | | | | | | |
| ------------- | --- | ------- | ----- | ------ | --------- | ------- | ------------------------ | --- | --- | -------------------------- | --- | --- | --- |
| | | | | | | | totheanalyticswarehouse. | | | Rowsareprojectedintoatyped | | | |
Aleader-electedprojectorbuildsadeterministicbundlecon-
| | | | | | | | usage table, | partitioned | | by day and | clustered | by | caller and |
| --- | --- | --- | --- | --- | --- | --- | ------------ | ----------- | --- | ---------- | --------- | --- | ---------- |
tainingpolicy,registry-derivedaccessdataandtheorganiza-
model,andaviewdeduplicatesonthegateway-generatedre-
| tion snapshot, | and | serves | it with | an entity | tag. Data-plane | | | | | | | | |
| -------------- | --- | ------ | ------- | --------- | --------------- | --- | --- | --- | --- | --- | --- | --- | --- |
questID,whichclientscannotspoof.Fromtheloggingagent
replicas poll conditionally and hot-reload. A failed reload onward, delivery is at-least-once with read-time deduplica-
| keeps the | last good | revision, | and | an | engine whose | initial | | | | | | | |
| ----------------------------------- | --------- | --------- | --- | --- | ---------------- | ------- | --------------------- | --------- | ----- | ------------------------------ | --------- | --- | ------- |
| | | | | | | | tion,barringnodeloss. | | | Oncealinereachesstandardoutput | | | |
| bundlefailstocompilerefusestoserve. | | | | | APIresponsesdis- | | | | | | | | |
| | | | | | | | it sits on | node disk | under | the container | runtime’s | | manage- |
tinguish the saved revision from the enforced revision, so ment,soacrashofthegatewayprocessafterwardsdoesnot
administratorscanseewhenachangeislive.
| | | | | | | | loseit. Theresiduallosswindowisbounded: | | | | | ahardproxy | |
| --- | --- | --- | --- | --- | --- | --- | --------------------------------------- | --- | --- | --- | --- | ---------- | --- |
Audit. Every mutation writes an append-only activity crash can lose at most one flush interval of buffered entries
record;policychangeswriteitinthesametransactionasthe (below), and a node failure can lose seconds of unshipped
change. Records can be filtered by category, outcome, ac- agentbuffer. Thisreplacedtheunbounded,silentlossmodes
torandrole;platformadministratorsseeallrecords,andBU oftheearlierin-processqueue. Theschemaisadditive-only
administratorsseetheirownbusinessunits. andversioned,sonewfieldsrequirenoinfrastructurechange.
Theserecords,oneperrequestwithtokensandcostbycom-
| 11 Telemetry, | | | Metering | and | | | | | | | | | |
| ------------- | --- | --- | -------- | --- | --- | --- | ------- | ---------------- | --- | --------- | ---------- | -------- | --- |
| | | | | | | | ponent, | drive chargeback | | and usage | analytics. | Appendix | C |
liststhefields.
Attribution
| | | | | | | | Implementation | | details | that mattered. | | First, | Envoy |
| --- | --- | --- | --- | --- | --- | --- | -------------- | --- | ------- | -------------- | --- | ------ | ----- |
Metrics and traces. Envoy’s native statistics (upstream silentlydropsdynamicmetadatareturnedbyanext_procser-
health, circuit breakers, connections, latency and response vice unless the filter allow-lists the receiving namespaces.
classes) and application metrics (requests, latency, token Without the allow list, every metadata-derived column is
counts by kind, cache reads and writes, cost, rate-limit null,andonlyanend-to-endtestagainstarealEnvoycaught
events,andschema-driftcounts)areexportedthroughOpen- it. Second, Envoy is configured to flush access-log entries
11
everyfewsecondsforstreamsthatarestillopen,ratherthan get keeps all but one of the minimum replicas avail-
onlywhenastreamcloses,whichboundswhatahardproxy ableduringvoluntarydisruptions,andtheproxytierau-
crash can lose on a long stream at negligible I/O cost. Pe- toscaleshorizontallyineachregion.
riodic entries preserve the record and its attribution; usage 5. The gRPC services drain the same way: they report
iscarriedbytheentrywrittenwhenthestreamends. Third, NOT_SERVING on their health endpoint and then stop
Envoy is configured not to preserve client-supplied request gracefully within a shorter grace period of their own.
IDs,soaclientcannotpoisonthededuplicationkey. Their rollouts surge a full replica set with zero unavail-
Context-engineering signals. The Wasm filter reports ability, because a fail-closed dependency must finish
which tools a model actually invoked. Combined with the drainingbeforeitstops.
| | | | | | | | | With this | sequence | we | have | deployed | more | than | ten times |
| -------------- | --------- | --- | --------- | --- | -------- | -------- | --- | --------- | -------- | --- | ---- | -------- | ---- | ---- | --------- |
| tools a client | declared, | | this lets | the | platform | identify | MCP | | | | | | | | |
serversandtoolsthatauserkeepsdeclaringacrossrequests duringlivevoicetraffic,withzerodroppedcalls,whilesus-
tainingmorethan2,000concurrentcalls.
| within a | 24-hour | session | window | but | that the | model | never | | | | | | | | |
| -------- | ------- | ------- | ------ | --- | -------- | ----- | ----- | --- | --- | --- | --- | --- | --- | --- | --- |
invokes,andreturnper-userrecommendationswiththeesti- Regional placement and failover. The gateway runs in
matedtokensavingsperrequestofremovingthem.
| | | | | | | | | multiple | regions | with | disaster | recovery | and | load | balancing. |
| --- | --- | --- | --- | --- | --- | --- | --- | -------- | ------- | ---- | -------- | -------- | --- | ---- | ---------- |
Whereadata-residencyendpointisconfiguredforavendor,
12 Reliability and Real-Time Traffic itisthedefault. Automaticfallbacktoanotherendpointisa
singleretry,limitedtoconnection-levelfailuresandservice-
| Voice changed | | the problem. | | Our | previous | gateway | was | | | | | | | | |
| ------------- | --- | ------------ | --- | --- | -------- | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
unavailableresponses;othererrorsarereturnedtotheclient
builtforshortrequest–responseexchangesbehindarequest-
ratherthanretried,whichkeepsretriesfromamplifyingload
orientedL7loadbalancer.Chattrafficsuitedit,becauseeach
duringavendorincident.
| request released | | its resources | | almost | immediately. | A | voice | | | | | | | | |
| ---------------- | --- | ---------------------------------- | --- | ------ | ------------ | --- | ----- | --------- | ------- | --- | --------- | --- | ------- | ------- | --- |
| | | | | | | | | Migration | without | a | big bang. | The | gateway | reached | its |
| callisdifferent: | | oneWebSocketheldforthewholeconver- | | | | | | | | | | | | | |
sationandstreamingaudioinbothdirections. Twothousand currentformwithoutaskinganyteamtochangeitsintegra-
| | | | | | | | | tion. First,Envoywasplacedinfrontoftheexistingservice, | | | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | ------------------------------------------------------ | --- | --- | --- | --- | --- | --- | --- |
concurrentcallsofupto30minuteseachpinnedper-session
| | | | | | | | | so nothing | behind | it changed. | | Then | authentication, | | creden- |
| ---------------- | ------- | ------------ | ----------- | ------ | -------- | ------------- | ---- | --------------- | ------------ | ----------- | ----- | ---- | --------------- | ----- | -------- |
| resources | for the | life | of the call | in a | service | sized for | sub- | | | | | | | | |
| | | | | | | | | tial brokering, | translation, | | spend | and | policy | moved | into the |
| second requests. | | In addition, | | a load | balancer | that balances | | | | | | | | | |
newdataplaneonecapabilityatatime,behindthesameen-
| requests | is a poor | fit for | long-lived | connections. | | Envoy | re- | | | | | | | | |
| ----------- | --------- | ---------- | ---------- | ------------ | ------------ | -------------- | ------ | ------------------------------------------- | ------------- | --- | --- | ------------ | -------------- | ----------- | --- |
| | | | | | | | | trypoint,followingastrangler-figpattern[8]. | | | | | | Progresswas | |
| solves both | problems: | | a call | becomes | a connection | | multi- | | | | | | | | |
| | | | | | | | | measured | continuously, | | and | the previous | implementation | | is |
| plexed on | Envoy’s | per-worker | | event | loops, | and connection | | | | | | | | | |
beingretiredcapabilitybycapability.
lifecycleisafirst-classproxyconcept.
| A connection-oriented | | | path. | Real-time | routes | are | tuned | | | | | | | | |
| ---------------------- | --- | --- | ------------------------------- | --------- | ------ | --- | ----- | ------------- | --- | --- | --- | --- | --- | --- | --- |
| | | | | | | | | 13 Evaluation | | | | | | | |
| forlong-livedsessions: | | | framesareforwardedastheyarrive, | | | | | | | | | | | | |
with no body buffering and no request timeout, and an idle 13.1 Gatewayoverhead
timeoutreclaimsabandonedconnections.Real-timesessions
| | | | | | | | | Method. | Wemeasuredthelatencythegatewayaddswitha | | | | | | |
| ------------ | --- | -------- | ------ | --- | --------- | ------- | --- | ------- | --------------------------------------- | --- | --- | --- | --- | --- | --- |
| are recorded | in | the same | access | log | and proxy | metrics | as | | | | | | | | |
other traffic, andreal-time model endpoints areavailable in Locustloadgenerator(userrampof500users/s, thinktime
morethanoneregion. 0.5–1s)againstasyntheticupstreamwithafixed110msla-
| | | | | | | | | tency. Each | instance | | has 4 | vCPU | and 8 | GiB, split | evenly |
| -------------------------------------- | ----------- | -------- | -------------- | --- | -------------- | ---- | ------- | ----------- | --------- | --------- | ------- | ------------------- | ----- | ---------- | ------ |
| Graceful, | drain-aware | | deployments. | | Pods | move | for or- | | | | | | | | |
| | | | | | | | | between | one Envoy | pod | and | one control-service | | pod. | Load |
| dinaryreasons: | | deploys, | autoscalingand | | nodeupgrades. | | A | | | | | | | | |
| | | | | | | | | steps use | 1,000 | and 2,000 | virtual | users. | Every | request | car- |
| chatrequestthatdiesinarolloutisaretry; | | | | | avoicecallthat | | | | | | | | | | |
diesisacustomerhunguponmid-sentence. Ourshutdown riesauniquepayload, andeachloadsteprunsforfivemin-
| | | | | | | | | utes. Overhead | | is the | time | from the | start | of the | client re- |
| --- | --- | --- | --- | --- | --- | --- | --- | -------------- | --- | ------ | ---- | -------- | ----- | ------ | ---------- |
sequenceisdesignedaroundthelongestcall:
| | | | | | | | | quest to | the start | of the | upstream | response, | | minus | the up- |
| ------------- | --- | ---------- | --- | ----- | --------- | ------- | --- | -------------- | --------- | ---------- | -------- | --------- | -------- | ----- | --------- |
| 1. A pre-stop | | hook marks | the | proxy | unhealthy | through | its | | | | | | | | |
| | | | | | | | | stream service | | time, both | recorded | | by Envoy | at | microsec- |
admininterface,sotheloadbalancerstopssendingitnew
| | | | | | | | | ond resolution; | | it therefore | measures | | the request | | path up to |
| --- | --- | --- | --- | --- | --- | --- | --- | --------------- | --- | ------------ | -------- | --- | ----------- | --- | ---------- |
connectionswhileexistingonescontinue.
2. The hook then polls the listener’s active downstream theupstreamcallandexcludesresponse-bodywork. Weran
| | | | | | | | | two and | four instances | | with | 1,000 | and 2,000 | virtual | users |
| ---------- | --- | ----- | ------------ | --- | ------- | ------- | ----- | ------- | -------------- | --- | ---- | ----- | --------- | ------- | ----- |
| connection | | count | periodically | and | returns | as soon | as it | | | | | | | | |
(about1,100and2,300requestspersecond)againsttheuni-
reacheszero,orshortlybeforethegraceperiodends.
3. The termination grace period (one hour) is longer than fiedchat-completionspath.
thelongestexpectedcall,becauseaconversationcanlast Results. Table 5 and fig. 6 show two results. First, over-
30minutesandholdingapodopenispreferabletocut- headremainedstableoverthetestedrange: doublingthere-
tingoffacustomer. quest rate on two instances changed no percentile, and on
4. Rollouts add a new proxy pod before removing an old fourinstancesthe2,000-userfigureswerewithin1msofthe
one(surgeone,zerounavailable),apoddisruptionbud- 1,000-userfigures. Second,addinginstancesdidnotchange
12
| | | | Overhead(ms) | | | | | Measure | | | | | | | Value |
| --- | --- | --- | ------------ | --- | --- | --- | --- | ------- | --- | --- | --- | --- | --- | --- | ----- |
Instances RPS p50 p95 p99 Failed Developersusingthegateway 12,000+
| | | | | | | | | Productionusecases | | | | | | | hundreds |
| --- | --- | ----- | --- | --- | --- | --- | --- | -------------------------------------- | --- | --- | --- | --- | ---------------------- | ------------- | -------- |
| | 2 | 1,137 | 4 | 10 | 11 | 0 | | | | | | | | | |
| | | | | | | | | Requestsperday | | | | | | | millions |
| | 2 | 2,257 | 4 | 10 | 11 | 0 | | | | | | | | | |
| | | | | | | | | Tokensperday(alltraffic) | | | | | | ∼200billion | |
| | 4 | 1,138 | 4 | 10 | 11 | 0 | | | | | | | | | |
| | | | | | | | | Models/modelfamilies/providers | | | | | 100+/multiple/multiple | | |
| | 4 | 2,262 | 4 | 9 | 10 | 0 | | | | | | | | | |
| | | | | | | | | Peakthroughputfromasingleworkload | | | | | | 60,000req/min | |
| | | | | | | | | Concurrentreal-timevoicecallssupported | | | | | | | 2,000+ |
Table 5: Gateway overhead (time to the start of the upstream response, Deploysduringlivevoicetraffic;droppedcalls 10+;0
minus upstream service time) with a synthetic 110ms upstream. RPS: Gatewayprompt-cachehitrate(bestworkloads) upto96%
achievedrequestspersecond. Availabilitytobusiness(ATB) 99.999%
| | | 2inst@1.1kRPS | | 2inst@2.3kRPS | | | | Table | 6: | | | | | | |
| --- | --- | ------------- | --- | ------------- | --- | --- | --- | ----- | ---------- | --------- | --- | ----------- | ------------- | ------- | ----- |
| | | | | | | | | | Production | operation | | (historical | point-in-time | figures | as of |
4inst@1.1kRPS 4inst@2.3kRPS September 2026, not service-level commitments). ATB is PayPal’s
availability-to-businessservice-levelindicator.
111111
| )sm(daehrevo | | | 101010 | | | | 10 | | | | | | | | |
| ------------ | --- | --- | ------ | --- | --- | --- | --- | ----------------------------------------- | ------------ | ------- | --------- | ------- | --------- | ----- | -------- |
| 10 | | | | 9 | | | | | | | | | | | |
| | | | | | | | | models | through | their | supported | entry | formats | with | no ven- |
| | | | | | | | | dorkeystomanageandnoper-teamprovisioning. | | | | | | | Gateway- |
| 5 | 4 4 | 4 4 | | | | | | | | | | | | | |
| | | | | | | | | level | prompt-cache | shaping | | reached | hit rates | of up | to 96% |
onagenticcodingworkloads,whichlowersinputcost,since
0
cachereadsarebilledatadiscount[3],andcanreducetime
| | p50 | | | p95 | | p99 | | | | | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
tofirsttoken,withnoclientchanges.
| Figure6:Overheadpercentilesbyinstancecountandload. | | | | | | Overheadwas | | | | | | | | | |
| -------------------------------------------------- | --- | --- | --- | --- | --- | ----------- | --- | ---- | -------------------- | --- | --- | --- | --- | --- | --- |
| | | | | | | | | 13.3 | Engineeringassurance | | | | | | |
essentiallyunchangedwhenloadorinstancecountdoubledwithinthetested
range.
| | | | | | | | | The | gateway | is security- | and | cost-critical, | | so we | treat its |
| --------- | ----- | ------------- | --- | ------------------ | --- | ---- | ------ | ----------- | ------- | ------------ | ------------ | -------------- | ---------- | ----- | --------- |
| | | | | | | | | correctness | | as a product | requirement. | | Continuous | | integra- |
| overhead, | which | is consistent | | with a per-request | | cost | rather | | | | | | | | |
than queueing delay at these rates. We did not test beyond tion runs the Go and Rust test suites with a coverage floor,
| | | | | | | | | the policy | | test suite | (which | requires | coverage | of every | pro- |
| --------- | -------- | --- | ----- | -------------- | --- | ---- | ------- | ---------- | ------ | ---------- | ------ | ----------- | -------- | -------- | ------ |
| 2,262 RPS | and make | no | claim | about behavior | | near | satura- | | | | | | | | |
| | | | | | | | | duction | policy | module), | bundle | validation, | | and the | cross- |
tion. Nostepproducedafailedrequest.
languagecontracttestsforthehandoff,togetherwithabuild-
| Threats | to validity. | | The benchmark | | measures | the | core | | | | | | | | |
| ---------------- | ------------ | ------------ | --------------- | --- | -------- | ----- | ------- | ------- | -------- | ---- | --------- | ----- | --------------- | --- | ------ |
| | | | | | | | | failing | coverage | test | for codec | field | classification. | | An ex- |
| request-handling | | path (caller | authentication, | | | model | resolu- | | | | | | | | |
tendedsuiteaddsdatabase-backedtestsforthecontrol-plane
| tion, credential | brokering | | and | request | rewriting) | against | a | | | | | | | | |
| ---------------- | ------------ | ---------- | --- | --------------- | ---------- | ------- | ---- | -------------------- | --- | ------------- | ----------------------------- | ------- | --- | -------- | ------ |
| | | | | | | | | stores | and | engine-parity | tests | between | the | previous | Go en- |
| synthetic | upstream; | production | | overhead | varies | with | the | | | | | | | | |
| | | | | | | | | gineandtheOPAengine. | | | Asyntheticupstreamservicelets | | | | |
| policies | and features | enabled | | for a workload. | | The | load | | | | | | | | |
end-to-endtestsandbenchmarksrunwithoutvendorspend.
| generator | ran inside | the | same | cluster, | and all | virtual | users | | | | | | | | |
| --------- | ---------- | --- | ---- | -------- | ------- | ------- | ----- | --- | --- | --- | --- | --- | --- | --- | --- |
Thedataplane,controlplaneandidentityservicescomprise
| shared one | credential, | so | caller | authentication | | was | served | | | | | | | | |
| ---------- | ----------- | --- | ------ | -------------- | --- | --- | ------ | --- | --- | --- | --- | --- | --- | --- | --- |
roughly45klinesofGoand7klinesofRust,withabout35k
| from the | control | service’s | in-process | cache. | The | metric | ex- | | | | | | | | |
| -------- | ------- | --------- | ---------- | ------ | --- | ------ | --- | --- | --- | --- | --- | --- | --- | --- | --- |
furtherlinesofGotests.
cludesresponse-bodyprocessing,whichincludesstreaming
translation.Wemakethemethodologyexplicitsothatothers
| | | | | | | | | 14 | Lessons | Learned | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | ------- | ------- | --- | --- | --- | --- | --- |
canreproduceit.
| | | | | | | | | L1. | Wrap, | don’t | rewrite. | Putting | Envoy | in front | of a |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | ----- | ----- | -------- | ------- | ----- | -------- | ---- |
13.2 Productionoperation
workingserviceandmovingcapabilitiesbehinditone
Table 6 summarizes production operation as of September atatimerequirednoclientmigrations,withnoregres-
2026.Allfiguresarehistorical,point-in-timeaggregates,not
| | | | | | | | | | sions | observed | during | the monitored | | rollout. | Measur- |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | ----- | -------- | ------ | ------------- | --- | -------- | ------- |
service-levelcommitments. ingtheshareoftrafficonthenewpathmadeprogress
| Effect of | centralization. | | | Several | outcomes | were | not | | visible. | | | | | | |
| --------- | --------------- | --- | --- | ------- | -------- | ---- | --- | --- | -------- | --- | --- | --- | --- | --- | --- |
achievable under direct integration. Governed AI traffic L2. Split control from data by latency profile. Out-of-
now flows through one mediated entry point, and model processforI/O-bounddecisionsandin-processforper-
callscarryper-userorper-application,per-platformandper- byte work let us be strict and fast at the same time.
workflow attribution and a priced usage record. Model The same proxy foundation is what made voice scale:
choice is configuration: switching the vendor or region be- long-livedconnectionsaremultiplexedonEnvoy’sper-
hind a model requires no application change, and onboard- workereventloopsratherthanpinningper-sessionre-
| ing a new | model | or changing | | a price | is a | validated | data | | sources. | | | | | | |
| --------- | ----- | ----------- | --- | ------- | ---- | --------- | ---- | --- | -------- | --- | --- | --- | --- | --- | --- |
change. Budgets are organization-aware, shadow-testable L3. Treattheconnectionlifecycleasafirst-classdesign
and retroactive within their window, and users are typically input. An HTTP request can be handed off and for-
warnedbeforetheyreachalimit. Engineersreachapproved gotten; a WebSocket must be held and then drained.
13
Upgrade,stream,idleanddrainshapedmostofourre- Work in progress includes: a single credential for reach-
liabilitydecisions. ingeverymodel,withautomaticmodelfallbacks;additional
L4. Make models, prices and policy data, and validate providers;extendingthesameentrypointtoMCPtoolcalls
themlikecode. Data-drivenconfigurationissafeonly and autonomous agents, with identity, policy and audit ap-
withstrictvalidation.Agatewaythatrefusestostarton plied to every action; further safety and quality controls at
badconfigurationissaferthanonethatstartsandroutes the gateway; cost- and latency-aware routing and semantic
incorrectly. caching, buildingonworksuchas[4,16]; andricher, more
L5. Storeusageatthefinestgrain,keyedbyimmutable proactivespendinsightsforowners.
| identifiers, | | andmakeeverybudgetaview. | | | | Thisde- | | | | | | |
| --------------------------------------------- | --- | ------------------------ | --- | --- | --- | ------- | --- | ------------- | --- | --- | --- | --- |
| cisionremovedtheneedfordatamigrationswhenbud- | | | | | | | | 16 Conclusion | | | | |
getsortagschange,enabledrulesthatapplytothewin-
dowtodate,andmade“whatwouldthisruledonow?” The Cosmos LLM Gateway shows that, in one large enter-
prise,theproblemsofLLMaccess(credentialsprawl,format
answerable.
L6. Keep decisions pure. Separating gather, decide and lock-in, ungoverned spend and uneven governance) could
effectmadepolicytestableandpreviewable,andmade be addressed at a single layer while adding a median of a
| | | | | | | | | few milliseconds | of measured | request-path | | overhead (sec- |
| --- | --- | --- | --- | --- | --- | --- | --- | ---------------- | ----------- | ------------ | --- | -------------- |
itpossibletomigrateenginesunderaparityharness.
L7. Shipgovernanceinshadowmode,andwarnbefore tion 13) for the developers and products that depend on it.
youblock. Always-ontrackingwithseparateenforce- The architecture rests on a few choices: governance out of
ment switches let new controls prove themselves on process and translation in process; models, prices and pol-
real traffic before they blocked anyone, and threshold icy as validated data; spend scopes as views over counters
keyedbyimmutablemodelidentifiers;purepolicydecisions
| reminders | | gave users | advance | notice | before | limits | ap- | | | | | |
| --------- | --- | ---------- | ------- | ------ | ------ | ------ | --- | --- | --- | --- | --- | --- |
plied. distributed as versioned bundles and introduced in shadow
L8. Anchor policy to stable identities. Rules that name mode,withusersnormallywarnedbeforetheyareblocked;
andadurable,pricedusagerecordwrittenwhereitishardest
organizationleadersratherthanteamlistssurvivereor-
ganizationswithoutedits. tolose. Viewedthisway,anenterpriseLLMgatewayisnei-
| | | | | | | | | therjustarouternorjustapolicyengine. | | | | Itistheplatform |
| ------- | ----------- | ------ | ----- | --- | ------------- | --- | -------- | ------------------------------------ | --- | --- | --- | --------------- |
| L9. Put | the durable | record | where | | it is hardest | | to lose. | | | | | |
Movingattributionfromanin-applicationqueuetothe layer that absorbs vendor multiplicity, credential variability
proxy’saccesslogreplacedsilent,unboundedlosswith andformatheterogeneityintoonegovernedsubstrate,sothat
adeveloperwritesagainstoneAPI,presentsoneidentity,and
asmall,boundedlosswindowandmadeusageasystem
ofrecord. reachestherightmodelthroughitssupportedentryformats
withgovernance,attributionandobservabilityappliedbyde-
| L10. Routingisnotthedifferentiator. | | | | | Off-the-shelfprox- | | | | | | | |
| ----------------------------------- | --- | -------------------------- | --- | --- | ------------------ | --- | ------ | ------ | --- | --- | --- | --- |
| iesroutewell. | | Whatourorganizationneeded, | | | | | andthe | fault. | | | | |
off-the-shelfproxyweevaluateddidnotprovideatthe
Acknowledgments
| time, | was | real-time | support, | identity-aware | | authoriza- | | | | | | |
| ----- | ------------------ | --------- | -------- | -------------- | --- | ------------- | --- | --- | --- | --- | --- | --- |
| tion, | organization-aware | | spend | control | | and workload- | | | | | | |
WethankthePayPalCosmosAIplatformengineering,prod-
identity-basedcloudcredentials.
| | | | | | | | | uct and program | teams, | and the many | internal | teams who |
| --- | --- | --- | --- | --- | --- | --- | --- | --------------- | ------ | ------------ | -------- | --------- |
adoptedthegatewayearlyandshapeditwiththeirfeedback.
| 15 Limitations | | | and | Future | Work | | | | | | | |
| -------------- | ----------- | ----------- | ------------ | ------ | ----------- | ---- | ----- | ------------------------------------ | --------------------- | ----------- | ------------------ | ---------- |
| | | | | | | | | All operational | figures are | historical, | point-in-time | aggre- |
| | | | | | | | | gates; they | are not service-level | commitments | | and do not |
| Several | limitations | remain. | Centralizing | | access | also | con- | | | | | |
| | | | | | | | | constituteforward-lookingstatements. | | | Thearchitecturede- | |
| centrates | risk: | the gateway | brokers | | credentials | for | every | | | | | |
scribedhereprovidestechnicalcontrolsonlyanddoesnotby
| provider, | so it is | a high-value | target | whose | own | access, | se- | | | | | |
| --------- | -------- | ------------ | ------ | ----- | --- | ------- | --- | ---------------- | ------------------- | --- | ----------- | ----------- |
| | | | | | | | | itself establish | legal or regulatory | | compliance. | Product and |
crethandlingandchangecontrolmustmeetthestandardof
projectnamesaretrademarksoftheirrespectiveowners.
| themostsensitiveworkloaditserves. | | | | Ourexperiencecomes | | | | | | | | |
| --------------------------------- | ----- | ------------- | --- | -------------------- | ------------ | --- | --- | ---------- | --- | --- | --- | --- |
| from one | large | organization, | and | other | environments | | may | References | | | | |
| weighthesetrade-offsdifferently. | | | | Ouroverheadbenchmark | | | | | | | | |
covers the core request path, and production overhead de- [1] AgentRouterAuthors.AgentRouter(formerlyEnvoyAIGateway):
pends on the policies and features enabled for a workload. Dataplaneandtrafficflow.https://theagentrouter.ai/docs/
Data-loss prevention relies on a learned detector whose ac- concepts/architecture/data-plane, 2026. Agentic AI Foun-
dationproject.AccessedOctober2026.
curacy varies by entity type, so it reduces rather than elim- [2] agentgatewayAuthors.agentgateway:AnAI-nativeproxyforagents,
inates exposure. Budget thresholds, notification points and toolsandmodels.https://agentgateway.dev,2026.AgenticAI
Foundationproject.AccessedOctober2026.
data-protectionpoliciesareorganization-specificandrequire
| | | | | | | | | [3] Anthropic. | Prompt caching. | https://platform.claude.com/ | | |
| --- | --- | --- | --- | --- | --- | --- | --- | -------------- | --------------- | ---------------------------- | --- | --- |
ongoingtuningagainstrealusage;themechanismsdescribed
| | | | | | | | | docs/en/build-with-claude/prompt-caching, | | | | 2026. Ac- |
| --- | --- | --- | --- | --- | --- | --- | --- | ----------------------------------------- | --- | --- | --- | --------- |
heremakethattuningsafe,butdonotremovetheneedforit.
cessedOctober2026.
14
| [4] LingjiaoChen, | | MateiZaharia, | andJamesZou.FrugalGPT:Howto | | | |
| ----------------- | --- | ------------- | --------------------------- | --- | --- | --- |
uselargelanguagemodelswhilereducingcostandimprovingperfor-
mance.arXivpreprintarXiv:2305.05176,2023.
| [5] Cloudflare. | AI | Gateway | documentation. | https://developers. | | |
| --------------- | --- | ------- | -------------- | ------------------- | --- | --- |
cloudflare.com/ai-gateway/,2026.AccessedOctober2026.
[6] EnvoyProjectAuthors.Envoyproxydocumentation.https://www.
envoyproxy.io/docs,2026.AccessedOctober2026.
| [7] Envoy | Project | Authors. | External | processing | filter | (ext_proc). |
| --------- | ------- | -------- | -------- | ---------- | ------ | ----------- |
https://www.envoyproxy.io/docs/envoy/latest/
configuration/http/http_filters/ext_proc_filter,
2026.AccessedOctober2026.
| [8] Martin | Fowler. | Strangler | | fig | application | (origi- |
| ---------- | ------- | --------- | ------------------------------- | --- | ----------- | ------- |
| nal | | text). | https://martinfowler.com/bliki/ | | | |
OriginalStranglerFigApplication.html,2004.
| [9] Google. | Gemini | API: Thought | signatures. | | https://ai.google. | |
| ---------------------------------------- | ------ | ------------ | ----------- | --- | ------------------ | --- |
| dev/gemini-api/docs/thinking#signatures, | | | | | 2026.Accessed | |
October2026.
[10] M.Jones.JSONwebkey(JWK).RFC7517,IETF,2015.
[11] M.Jones,J.Bradley,andN.Sakimura.JSONwebtoken(JWT).RFC
7519,IETF,2015.
| [12] WoosukKwon, | | ZhuohanLi, | SiyuanZhuang, | | YingSheng, | Lianmin |
| ---------------- | --- | ---------- | ------------- | --- | ---------- | ------- |
Zheng,CodyHaoYu,JosephE.Gonzalez,HaoZhang,andIonStoica.
Efficientmemorymanagementforlargelanguagemodelservingwith
PagedAttention.InProceedingsofthe29thSymposiumonOperating
SystemsPrinciples(SOSP’23),pages611–626,2023.
| [13] Microsoft. | AI | gateway | capabilities | in | Azure | API Manage- |
| ------------------------------------------ | --- | ---------------------------------------- | ------------ | --- | ----- | ----------- |
| ment. | | https://learn.microsoft.com/en-us/azure/ | | | | |
| api-management/genai-gateway-capabilities, | | | | | | 2026. |
AccessedOctober2026.
| [14] Model | Context | Protocol | Authors. | Model | context protocol | speci- |
| ---------- | ---------------------------------------------- | -------- | -------- | ----- | ---------------- | ------ |
| fication. | https://modelcontextprotocol.io/specification, | | | | | |
2026.AccessedOctober2026.
| [15] NVIDIA. | Dynamo: | A datacenter-scale | | distributed | | inference serv- |
| ------------ | ------- | ------------------ | --- | ----------- | --- | --------------- |
ingframework.https://github.com/ai-dynamo/dynamo,2026.
AccessedOctober2026.
| [16] Isaac | Ong, Amjad | Almahairi, | Vincent | Wu, | Wei-Lin | Chiang, Tian- |
| ---------- | ---------- | ------------ | ------- | -------------- | ------- | ------------- |
| hao | Wu, Joseph | E. Gonzalez, | M. | Waleed Kadous, | and | Ion Stoica. |
RouteLLM:LearningtorouteLLMsfrompreferencedata.InInter-
nationalConferenceonLearningRepresentations(ICLR),2025.
[17] OpenPolicyAgentAuthors.Openpolicyagent:Policy-basedcontrol
| for cloud | native | environments. | https://www.openpolicyagent. | | | |
| --------- | ------ | ------------- | ---------------------------- | --- | --- | --- |
org/docs,2026.AccessedOctober2026.
| [18] OpenTelemetry | | Authors. | OpenTelemetry | specification. | | https:// |
| ------------------ | --- | -------- | ------------- | -------------- | --- | -------- |
opentelemetry.io/docs/specs/otel/,2026.AccessedOctober
2026.
[19] RuomingPang,RamónCáceres,MikeBurrows,ZhifengChen,Pratik
| Dave, | Nathan | Germer, Alexander | | Golynski, | Kevin | Graney, Nina |
| ----- | ------ | ----------------- | --- | --------- | ----- | ------------ |
Kang,LeaKissner,JeffreyL.Korn,AbhishekParmar,ChristinaD.
| Richards,andMengzhiWang.Zanzibar: | | | | Google’sconsistent,global | | |
| --------------------------------- | --- | ---------- | ----------- | ------------------------- | ----------- | ------ |
| authorization | | system. In | Proceedings | of the | 2019 USENIX | Annual |
TechnicalConference(USENIXATC’19),pages33–46,2019.
| [20] Portkey | AI. | Gateway: | An open-source | | AI gateway. | https: |
| -------------------------------- | --- | -------- | -------------- | ----- | ----------- | ------- |
| //github.com/Portkey-AI/gateway, | | | | 2026. | Accessed | October |
2026.
[21] Proxy-WasmAuthors.WebAssemblyforproxies(ABIspecification).
| https://github.com/proxy-wasm/spec, | | | | | 2026.AccessedOcto- | |
| ----------------------------------- | --- | --- | --- | --- | ------------------ | --- |
ber2026.
[22] N.Sakimura,J.Bradley,M.Jones,B.deMedeiros,andC.Mortimore.
OpenIDConnectcore1.0.OpenIDFoundation,2014.
| [23] PayPal | AITech | Cosmos | AI Team. | One | connection. | 2,000 |
| ----------- | ------ | --------- | --------- | --- | ------------- | ----- |
| tools. | PayPal | Developer | Community | | Blog, October | 2026. |
https://developer.paypal.com/community/blog/One_
connection_2000_tools/.
| [24] Lianmin | Zheng, | Liangsheng | Yin, | Zhiqiang | Xie, Chuyue | Sun, Jeff |
| ----------------- | ------ | ------------- | ---- | -------------------- | ----------- | ----------- |
| Huang, | Cody | Hao Yu, Shiyi | Cao, | Christos | Kozyrakis, | Ion Stoica, |
| JosephE.Gonzalez, | | ClarkBarrett, | | andYingSheng.SGLang: | | Effi- |
cientexecutionofstructuredlanguagemodelprograms.InAdvances
inNeuralInformationProcessingSystems(NeurIPS),2024.
15
A Registry Excerpt
Abridged and sanitized. The model identifier is illustrative and prices are placeholders (USD per million tokens). Provider,
credentialandpathconfigurationareomitted.
families: { chat: {}, responses: { canonical_format: openai_responses }, embeddings: {} }
wire_formats: { anthropic_messages: { family: chat }, openai_chat: { family: chat },
anthropic_vertex: { family: chat }, gemini_generate_content: { family: chat },
openai_responses: { family: responses } }
converters:
- { from: anthropic_messages, to: openai_chat }
- { from: anthropic_messages, to: anthropic_vertex }
- { from: anthropic_messages, to: openai_responses }
- { from: anthropic_messages, to: gemini_generate_content }
- { from: openai_chat, to: anthropic_vertex }
- { from: openai_chat, to: openai_responses }
- { from: openai_chat, to: gemini_generate_content }
models:
claude-opus-x: # illustrative model id
brand: anthropic
entry_formats: [anthropic_messages, openai_chat]
aliases: [claude-opus-x-latest]
tags: [all, claude, opus, proprietary, frontier]
deployments:
- id: opus-x@cloud
provider: <cloud-ai-platform>
region: global
deployment_format: anthropic_vertex
priority: 1
pricing: { unit: usd_per_mtok, input: <in>, output: <out>,
cache_read: <cr>, cache_write: <cw>, cache_write_1h: <cw1h>,
threshold_tokens: <n>, # optional long-context tier
long: { input: <in2>, output: <out2>, cache_read: <cr2>, cache_write: <cw2> } }
B Spend Rule Language
Eachruleholdsoneormoreentries;eachentryreadsasonesentence:whenauser’sspendinascopecrossesathresholdwithin
a window, do these actions. Identifiers, weights, amounts and template names are illustrative placeholders, not production
values. scope is a registry tag or a single model identifier and is resolved at evaluation time (section 9.2). Rules default to
enabled,becauseacapsomeoneforgottoenablesilentlydoesnothing,whichistheworsefailureforcostcontrol. Whetheran
enabledrulecanblockisgovernedseparatelybytheenforcementmode(section10.4). Thefirstrulebelowshowsthereminder
pattern: awarningemailat90%ofanallowance,thenablockandasecondemailat100%.
rules:
- id: frontier-family-monthly # stable id; surfaces in metrics
weight: 60 # precedence (higher first)
matches:
- match: { tiers: [normal], scope: opus, window: monthly, at_or_above_usd: <0.9 x T> }
actions:
- { type: email, to: [user], subject: "You’re at 90% of your monthly limit", body_file: <template> }
- match: { tiers: [normal], scope: opus, window: monthly, at_or_above_usd: <T> }
actions:
- { type: email, to: [user], subject: "Monthly limit reached", body_file: <template> }
- { type: block } # stateless; clears when the window rolls
- id: daily-pause
weight: 50
matches:
- match: { scope: all, window: daily, at_or_above_usd: <D> }
actions:
- { type: pause, hours: <H> } # one-shot per (rule, window); self-lifts
- { type: message, level: warn, text: "Paused. Resumes {resume_time}." }
- { type: email, to: [user] }
- id: org-monthly-cap
weight: 100
matches:
- match: { tiers: [normal, pro], scope: all, window: monthly, leaders: [<org-leader>],
exempt_users: [<login>], at_or_above_usd: <B> }
actions:
- { type: block }
- { type: message, level: warn, text: "You have reached your organization’s monthly per-person limit." }
- id: weekly-leadership-awareness
weight: 40
matches:
- match: { tiers: [normal, pro], scope: all, window: weekly, at_or_above_usd: <W> }
actions:
- { type: email, to: [leadership] } # deferred digest after the window boundary
16
C Usage Record
Onerowperrequestinaday-partitionedwarehousetable,clusteredbycallerandmodel,readthroughaviewthatdeduplicates
ontherequestID.
Fieldgroup Fields Origin
Identityoftherequest requestid(gateway-generated),timestamp,traceid Envoy
Caller user,application,tier,platform/workflow/harnesstags(oneJSONobject) control(requestphase)
Routing requestedmodel,canonicalmodel,vendor,entryandupstreamformats,resourcetype control(requestphase)
Usage input,output,cache-read,cache-write(incl.extended-lifetime)tokens Wasmtrailers
Cost per-componentandtotalUSD;workflowspent/limitwhenpresent control(trailerphase)
Outcome status,responseflags,duration,timetofirstbyte,upstreamcluster Envoy
17