claude-news
blogArdent Performance Computing·21 Oct 2026·Jeremy Schneider

recovery_prefetch Considered Harmful and Causing Outages

A field report showing that recovery_prefetch on Postgres replicas can quietly cause outages, how it was diagnosed, and when to turn it off.

Read original on ardentperf.com Discussion
PostgreSQLdatabase-operationsreplicationpostmortem
Depth 5/5Score 4.86

Extract

Another story of something I saw in the field which surprised me. Turns out recovery_prefetch can cause outages. Something which has been happening quietly in the background all the time, but we didn’t notice until we looked close enough to see it. After a lot of work I discovered this was partly a case where someone had dropped an email to the postgres hackers list years ago “hey what about this” and everyone thought “nah it can’t be that bad” … and now here we are seeing production outages! Exhibit A: Feb 2021, while the feature was being developed, a year and a half before it shipped, Stephen Frost Issuing unnecessary prefetches for blocks we’ve already sent a prefetch for is arguably a concern even if FPWs are off but the benefit of doing the prefetching almost certainly will outweight that and mean that finding a way to address it is something we could certainly do later as a future improvement. I wouldn’t have any issue with that. Just doesn’t seem as clear-cut to me when thinking about the FPW-enabled case. Ultimately, if you, Andres and Munro are all not concerned about it and no one else speaks up then I’m not going to pitch a fuss over it being committed, but, as you said above, it seemed like a good point to raise for everyone to consider. https://www.postgresql.org/message-id/20210214231815.GO27507@tamriel.snowman.net Exhibit B: Mar 2025, clear report from the field, Michael Zhilin (Postgres Pro) One customer complained that physical standby has significant replay lag (few hours) on modern powerful bare metal servers. According to output of “top”, startup process and few kernel workers utilized CPU, mostly in kernel space. The incident has been resolved by disabling recovery_prefetch after quick analysis of perf and strace outputs. Actual XLog prefetching code doesn’t consider that shared buffer may be loaded from WAL FPI rather than from disk and makes fadvise calls in such cases. https://www.postgresql.org/message-id/9bdb428a-fc8e-4801-8fff-1ad0bc6fa76b%40postgrespro.ru Both of these emails were on the right track, but didn’t quite get the complete picture yet. First: FPI/FPW is ubiquitous today. While it’s very expensive in terms of performance, it’s the only way Postgres has of protecting from torn writes on OS restarts. As a result, the majority of production Postgres installations today (community oss version) enable this. Second: synchronous replication is becoming more common. There are really only two ways to completely protect …

Explained

What happened

Jeremy Schneider (ardentperf) shows that the Postgres setting recovery_prefetch can cause full write outages on clusters using synchronous replication. Reproduction on a small box (500 IOPS disk, 2 CPUs, 8 GB):

  1. Primary with synchronous replication to one standby, pgbench scale 640 (~10 GB), steady 5 TPS.
  2. Update 5% of pgbench_accounts, a burst of writes.
  3. Kill the walreceiver to simulate a network blip.
  4. TPS drops to zero and all writes hang for 30+ minutes (hours with bigger tables).

With recovery_prefetch = off (full page writes still on), the standby applies the same update in under 10 minutes and the long hang goes away. Repro: github.com/ardentperf-labs/walreceiver-outage-repro.

Why it works

Three things combine:

  • IOPS amplification. When the primary does large sequential I/O, prefetch on the replica turns it into 8 kB random reads, up to 16x more IOPS for the same bytes. It also issues fadvise calls for pages that will come from full page images (FPI) in the WAL, so the reads are useless. The standby sat at its 500 IOPS limit for 40 minutes while the primary did about 300 IOPS for 5.
  • The standby state machine. In xlogrecovery.c, a standby that loses streaming falls back to reading WAL from archive or pg_wal and does not switch back to streaming until it has caught up.
  • Synchronous commit needs a healthy standby. The primary blocks every commit until a sync standby confirms. Lagging standby that won't stream means a frozen primary.

Both problems had been flagged on pgsql-hackers (2021 and 2025) but nobody had connected them to outages.

The bigger pattern

This is a classic prefetch amplification bug: an optimization that is correct alone becomes harmful when it ignores where the data really comes from. It also shows how synchronous replication couples the primary's availability to the slowest replica's I/O.

What it means in practice

  • If you run Postgres with synchronous_standby_names, check recovery_prefetch on standbys (default try since PG 15). The author leans toward turning it off everywhere while FPW is on.
  • Prefer two standbys with quorum commit (ANY 1 (s1, s2)) over a single sync standby. One flaky link then can't stop writes.
  • Alert on replay lag and on a standby sitting in archive/pg_wal mode instead of streaming.
  • Large backfills, UPDATEs and VACUUM bursts are the trigger. Throttle them on clusters with sync replicas.