Sunday, 25 October 2026
Rebuilt for scale: storage, LLM serving and isolation at billion-user size
Top 10 picked from 40 candidates
- 1
OpenAI moves ChatGPT storage to co-managed multi-tenant Cosmos DB for 1B users
Habitat, the storage platform behind all OpenAI products, serves over 1 billion users across 40 regions with more than 500 PB of data. Part 2 covers scaling the Azure Cosmos DB layer: after swinging between single-tenant and multi-tenant setups, they landed on co-managed containers that keep noisy neighbors in check while giving fast global reads.
Why it matters. It is a rare first-hand account of the single vs multi-tenant tradeoff at extreme scale, with the reasoning, not just the end state. The same choice shows up in any shared data platform for video or AI products.blogOpenAI Engineering · 24 Oct 2026“Rapidly scaling online storage to serve over 1 billion ChatGPT users (…” - 2
vLLM hits 954K tokens/s per GPU on a GB300 NVL72 rack, within 6% of DeepSeek
Using prefill/decode disaggregation and wide expert parallelism, vLLM serves DeepSeek-V4.1-Flash at 954K tokens/s per GPU at P90 196 tokens/s per user, 7.9x its single-node result, at about 39 dollars per GPU-hour. Decode throughput lands within 6% of DeepInfer, DeepSeek own engine, and CPU overhead turned into a key bottleneck at rack scale.
Why it matters. It shows a general-purpose engine can match a model-specific one, and it lists which levers actually mattered: small prefill instances, wide-EP decode with load balancing, and cutting CPU overhead.blogvLLM Blog · 23 Oct 2026“Serving DeepSeek-V4.1-Flash at Scale: vLLM Matches DeepInfer and Scale…” - 3
Cloudflare stops pulling 128 MiB from origin for every video seek
The old range caching mapped each request to fixed-size slices, so one seek could fetch 128 MiB from origin and move it between data centers. The rebuilt Origin Range Requests fetches what is needed with no range size to tune, halving p99 time to first byte for R2 and cutting tiered cache traffic by about 6%.
Why it matters. Range requests are how video gets delivered, and fixed slice sizes waste bandwidth and disk in both directions. This is a clear design to copy for any CDN or video origin layer.blogCloudflare Blog · 24 Oct 2026“Origin Range Requests: Serving large files without range-request tunin…” - 4
Netflix adds dedicated-VM tasks to Titus without changing its container API
Titus runs millions of containers a week across tens of thousands of EC2 instances, mostly multi-tenant. For latency-critical services that shared-kernel isolation cannot protect, Netflix built single-tenant mode: each task gets its own VM while keeping the same API, deploy path and managed hosts. They compared microVMs, gVisor and plain EC2, and measured isolation instead of assuming it.
Why it matters. Noisy neighbors on shared kernels are a real tail-latency problem for streaming services. Offering isolation as a mode inside one platform is a pattern worth knowing.blogNetflix TechBlog · 23 Oct 2026“Single-tenant Titus: VM-level isolation on a multi-tenant container pl…” - 5
ChatGPT ads run on a post-trained LLM that does retrieval, ranking and auctions
Intent in chat develops over many turns, so OpenAI built stateful ad sessions and a post-trained LLM that calls tools for retrieval, engagement ranking, relevance checks and auctions. Ad selection runs in a separate service from answer generation. The product reached a 1 billion dollar revenue run rate in under 200 days.
Why it matters. It is a new shape for ads and recommendation: an LLM orchestrating the classic retrieve, rank and auction funnel over conversation state. It is directly relevant to recommender work in AI products.blogOpenAI Engineering · 21 Oct 2026“How we built ChatGPT’s novel ad stack for user trust and relevance” - 6
Eight trusted eMudhra intermediate CA keys are compromised, so any domain can be forged
On October 19 the private keys of eight widely trusted eMudhra intermediate certificates were found to be weak and compromised, which lets an attacker forge a certificate for any domain. Cloudflare patched trust stores, base images and clients first, saw no exploitation, and warns that updating an SDK may not change the trust store your HTTP library actually uses.
Why it matters. Services that call external APIs or origins may still trust these CAs through bundled or OS trust stores. It is worth auditing where your fleet actually gets its trust roots.blogCloudflare Blog · 21 Oct 2026“8 eMudhra certificates are compromised. Here's what you need to know” - 7
vLLM serves 1M-token contexts for GLM-5.3 on 16 H200s with context parallelism
With one prefill node and one decode node, vLLM splits prompts across GPUs by token (PCP) and shards the KV cache by token (DCP), while Hybrid HiSparse keeps decode KV in host memory. This gives 1.7 to 3.2x the per-user speed of data parallel at the same throughput, and it was the only setup tested that serves 1M-token contexts.
Why it matters. Agentic workloads keep growing context every turn, so KV memory and prefill become the limits. This shows why tensor and data parallelism fall short for sparse attention models.blogvLLM Blog · 23 Oct 2026“GLM-5.3 Optimizations, Part 2: PCP, DCP, and Hybrid HiSparse on 16 H20…” - 8
Postgres recovery_prefetch can stall replicas for hours under full-page writes
A field investigation found that recovery_prefetch issues useless fadvise calls for pages already restored from full-page images in the WAL, burning kernel CPU and causing hours of replica lag. The issue was raised on the hackers list in 2021 and dismissed, and it now hits common setups with full-page writes and synchronous replication.
Why it matters. Replica lag directly hurts failover and read scaling. If you run Postgres replicas, check this setting before it shows up as an outage.blogArdent Performance Computing · 21 Oct 2026“recovery_prefetch Considered Harmful and Causing Outages” - 9
Cloudflare cuts its AI code review cost 68% and makes reviews 2.7x faster
Cost per review had doubled to about 2.46 dollars as changes grew and reviewers moved to pricier models, even with a 97% cache hit rate, since tokens per review tripled. Code fixes gave about 40% of the savings and model and effort choices the rest, bringing reviews to 0.77 dollars and the median time from 4m09s to 1m32s.
Why it matters. High cache hit rates do not save you when token counts grow. It is a practical playbook for keeping agent workloads affordable as they scale.blogCloudflare Blog · 20 Oct 2026“Cutting our inference bill by 68% by being obsessive about optimizatio…” - 10
Netflix Quasar shows which repos call a library API before you break it
Quasar indexes symbol definitions and references across Netflix repositories, so a library owner can see exactly which consumers and versions use the symbols a change touches. That turns a generic compatibility warning into evidence for whether to release, deprecate or plan a migration, and it gives coding agents the same cross-repo context.
Why it matters. Knowing the real blast radius of a shared library change is a core staff-level problem in large codebases, and it matters more as agents make changes faster.blogNetflix TechBlog · 24 Oct 2026“Beyond API Compatibility: Building a Tool to Measure the Blast Radius …”