SBD: We Have a Filesystem at Home

I have unkind things to say about people who make their own pasta. Here, I’ll only use them as an example of how not to think when you’re building infrastructure. It’s unfairly easy to believe that the harder you work on something, the better it gets. But infra is about making the $1 bag of spaghetti: a simple thing that works well (and doesn’t require you to spend a week scraping flour out of the corners of your kitchen). When work on Sprites started, it was a new kind of infra for a new kind of product. So when we couldn’t find the right filesystem in the pasta section, we made our own. It was clever, and that’s exactly why we didn’t like it. This post introduces its replacement, the Sprites Block Device (SBD). It’s the bag of spaghetti you grab off the shelf, and the only difference you’ll notice is better reliability. It was always object storage Our stance on one core aspect of Sprites’ storage hasn’t changed from v1: the root of a Sprite’s disk is object storage, not the NVMe in whatever server the Sprite happens to be running on. That’s what lets a Sprite sleep, wake on a different server, and still find its files where it left them. The durable state of a Sprite is, more or less, a location in a bucket. We wrote about why , at length, with feeling. The thing about object stores is they can’t change part of an object. To change one byte, you have to replace the whole thing. That’s a bad cultural fit for Linux programs which rewrite a few bytes in the middle of a file all the time. Something has to handle that translation. At launch, that something was a heavily modded version of JuiceFS. In stock form, JuiceFS splits storage into the data (as chunks in the bucket) and the metadata (a map of which chunks make up which files) in a database. That’s not how we used it, though. Our hot rodded version ran a rewritten SQLite metadata backend, kept durable with Litestream. It stored exactly one file: a 100 GB disk image, mounted on a loop device with ext4 inside. Linux did all the ordinary ext4 things. JuiceFS just held the image. It worked fine. Sprites shipped on it, people made a ton of them, and checkpoints were fast because checkpointing was mostly just shuffling metadata around. But as their usage grew, cracks began to emerge. Manifests that lie A filesystem with a metadata database has two inseparable components: the chunks in the bucket, and the database that explains how those chunks fit together. …