The Polyglot's Dilemma: Conformance Testing a Dozen Specs in as Many Languages

MongoDB engineers juggling many implementations of many specs. At MongoDB, we develop a dozen client libraries (“drivers”) in a dozen programming languages. A few wrap shared cores written in C or Java, but mostly they’re rewrites in their target languages: the JavaScript Driver is pure JavaScript, the Rust Driver is pure Rust, and so on. Building the drivers this way is a big commitment. Lots of database companies just write their client in C and make thin wrappers in various languages. Our way is harder but, in my opinion, better. Our drivers are easier to install, their APIs are more idiomatic in each language, and they participate normally in each language’s runtime: its threading model or async loop, its garbage collection, etc. This is only possible because we keep developing native drivers in each language. But then, how do we ensure all our drivers’ behaviors and APIs are consistent? Or, that they’re appropriately consistent or inconsistent, according to the conventions of their target languages? Obviously, Python methods are snake_cased and Java methods are camelCased, but MongoDB drivers should have the same basic API in all languages, spelling aside. Node concurrency is async and C concurrency uses threads, but the meaning of causal consistency should be the same in all drivers. In Search of a Tolerable Test Language

I joined MongoDB as a Python driver engineer in November 2011. Looking back through my emails, I see that we were intensely debating how to standardize our drivers from the moment I arrived. Some engineers were writing Java Driver tests in Cucumber, a “behavior-driven development” syntax, like this: Scenario: Exclude a field from query results Given collection "users" contains documents """ {"_id": 1, "name": "alice", "password": "hunter2"} {"_id": 2, "name": "bob", "password": "swordfish"} """ When find documents by """ {"name": "alice"} """ And return fields """ {"password": false} """ Then the result should be """ {"_id": 1, "name": "alice"} """ My very first task at MongoDB was researching whether we could write test-runners for these Cucumber tests in Python, Ruby, and our other languages. I instantly learned that my second task was dealing with my colleagues’ revulsion to Cucumber’s syntax. One engineer wrote that if we replaced his handwritten tests with Cucumber, he’d “have no part in doing it or supporting the resulting disaster.” (I clarified that we weren’t going to replace tests, only complement …