Animations for System Builders: Metastability and Lost Writes

About Me My name is Marc Brooker. I like to build things that work, and do cool stuff. I like building big things. I also dabble in machining, welding, cooking, and skiing. I am an engineer at Amazon Web Services (AWS) in Seattle, where I work on agentic AI, especially safety and policy for agentic AI. Before that, I worked on EC2, EBS, databases, serverless, and serverless databases. All opinions are my own. Links My Publications and Videos @marcbrooker on Mastodon @MarcJBrooker on Twitter Is this blog written by AI? Pretty pictures help us learn, maybe? One of the things I love using LLMs for is helping me understand systems better. In the Strands Decider post I wrote about how I often ask ask me questions to check I understand this. The other technique I’m enjoying right now, and am delighted to see folks around me using too, is building animations. Animations can help us build an intuition for how and why systems work, or how and why they fail. Here’s one example: did you know that a Raft cluster can lose writes if even one of the cluster members has a storage layer that lies about fsync()? Your browser does not support the video tag. When I built Stability Sim, I wanted to help engineers understand metastable failures better. That worked pretty well on an abstract level (e.g. retries and queues make metastability really easy to trigger), but I still hear doubts that these behaviors happen in real systems. After all, those real systems are production-proven, and wouldn’t we have shaken out the behaviors by now if they existed? Let’s start with Postgres. Your browser does not support the video tag. In this example, captured from traces on real Postgres 16 running in a real MicroVM. Before a load spike, the database is easily able to handle its 20 queries-per-second load, thanks to the whole working set sitting in the page cache. But after the load spike, we’ve filled up a lot of the memory running queries (each takes 12MB), which now run slower because they’re served off the disk rather than the page cache. I like this example, because it shows that even simple, non-distributed systems have these behaviors. What about etcd? Your browser does not support the video tag. Again, we have load spike triggering the metastable state. Writes, in this case. The overloaded peers look like they’re unavailable, which triggers a storm of application-level resends. I like this example, because etcd is widely-used quality software. Operational experience …