Cutting our inference bill by 68% by being obsessive about optimization
Cloudflare cut the cost of its AI code reviewer by 68 percent, from about 2.46 dollars per review, through caching, model routing and prompt changes.
Extract
When we previously blogged about our AI code reviewer in April , an average review cost a palatable $1.19. By the end of August, that figure had shot up to about $2.46. We hadn't made any significant architectural changes: the coordinator was still handing work to the same set of specialist reviewers, and the same risk tiers decided how intensive of a review we gave. Around mid-September we tasked our agents with working out where those tokens were going and cutting them — aggressively. With code quality at high effort, the average review now costs just $0.77, down 68% from the $2.43 it averaged over our two most expensive weeks before we started. Our main goal was to cut costs, but as part of this work we also managed to optimize the reviewer, so that the median review now takes just 1m 32s instead of the 4m 09s it took before. About 40% of the savings came from changes and optimizations to the reviewer's code, and the rest from experimenting with which models we use and how much effort they put in. How a review doubled in cost The chart below shows the cost of an average review on each weekday since March. It holds the mix of review tiers fixed, so a day with more large reviews doesn't skew the data. The cost per review steadily rose from $1.22 (March) → $1.44 (June) → $1.74 (July) and finally, to $2.32 (August). A small part of that is that more reviews landed in the most expensive tier (59% of reviews in March and then 65% in August). We believe there’s a simple answer for why: models are getting much better, and much more capable. Naturally, engineers are producing bigger, more complex changes which means more of them fall into our “full review” tier. We found that most of the cost increase actually came from model changes over those months, though. At the start of August we moved every reviewer onto Anthropic models, which raised the cost per job from $1.38 to $2.03. We made this choice because we found that during our evals, Anthropic models performed better on this specific workload than other models. Our internal standards reviewer, Codex , ended up on Claude Opus 4.7 due to a mix of performance and reliability. That took it from about $0.25 per review to $0.88 in August, which made it more expensive than the coordinator agent itself. The cache hit rate also increased, going from 85.7% in April to 97% by September, while the number of tokens per review more than tripled, from 0.91 million to 2.99 million. A high hit rate on three …
Explained
What happened
Cloudflare cut the cost of its internal AI code reviewer from about $2.43 to $0.77 per review (68% less), and the median review went from 4m 09s to 1m 32s.
- Cost had doubled since spring with no architecture change. Bigger changes pushed more reviews into the top tier, and moving all reviewers to stronger models raised the per-job cost.
- Tokens per review tripled (0.91M to 2.99M). Spend was 43% cache writes, 34% cache reads, 23% output: most money went to context the model had seen before.
- Logging the first call of each session showed only 14 to 26% found a cached prefix, even though 11k to 21k tokens of tools and prompt were the same everywhere. The harness put the working directory and date in the same block as the shared prompt, so the only cache breakpoint sat after per-repo bytes.
- They split that block with a marker so the shared part got its own breakpoint, and removed anything repo-specific from in front of it. They also inlined a reference index that every session had spent three tool calls reading. One reviewer dropped from 15.9 to 9.6 calls per run.
- Smaller fixes: hide a tool sub-reviewers were not allowed to call (they tried in 57 to 76% of sessions), read files in parallel, skip reviewers with nothing new on re-reviews, stop the extra summary turn, and actually stop cancelled jobs (358 of 446 kept running).
- Three reviewers now send an identical first request. One writes the cache, the other two wait up to 20 seconds for its response to start streaming, then read it.
- About 40% of the saving came from code, the rest from model and effort choices.
Why it works
Prompt caching only works on an exact prefix, up to a cache breakpoint. A write costs 1.25x input price, a read 0.1x, and entries expire after five minutes. Agent sessions resend the whole conversation on every call, so cost grows with calls times context size. Any changing byte early in the prompt turns cheap reads into expensive writes, and every extra tool call re-reads everything.
The bigger pattern
This is classic cache key design. Put stable data first and volatile data last, the same way you would order a composite cache key or an HTTP response to keep it cacheable. The coordinated first request is a form of request coalescing to avoid a thundering herd of cache misses. None of this is new in caching, but most agent harnesses ignore it by default.
What it means in practice
- Log the cache hit rate of the first call in each session and hash the system prompt as sent. A low rate usually means something volatile sits too early.
- Keep dates, paths, user IDs and dynamic tool lists after the shared prefix.
- Count calls per session. Remove tools the agent cannot use, batch reads, and drop turns nobody reads.
- When parallel agents share context, let one warm the cache first.
- Make job cancellation real. Many CI and queue runners do not signal the running process, so poll your own status.