Software is slow because of bad design, not slow languages

Many people are saying that AI will make software faster by rewriting it in faster languages, but this isn’t generally true. Most programs cannot be meaningfully sped up by rewriting them in C or Rust. How can this be true? Some languages are orders-of-magnitude faster than others: for instance, C can be fifty times faster than Python. Won’t rewriting my Python program in C thus make it fifty times faster? No, it won’t. Software performance is usually dominated by I/O, not the speed of the language. Where language speed matters, most slow languages already have popular libraries that allow them to unlock C-like speed. In general, performance is a matter of design, not of language choice. Most real programs are I/O-bound Consider a simple CRUD website with a database. For any given request, 50% of the request time1 might be spent querying the database: to fetch the current user, the record being asked for, any relevant user memberships or settings, and so on. Rewriting that website in C can only speed up that other 50%, which puts a hard ceiling on the improvements you can make2. At best, you’ll make requests twice as fast (probably closer to 1.5x in practice). That’s not bad, but it’s nowhere near an order-of-magnitude speedup. In other words, most real programs are “I/O-bound”: they spend most of their time waiting for data to be input or output somewhere, as opposed to “CPU-bound” programs that spend most of their time actively doing things. I/O-bound programs don’t just have to rely on a database. Often programs have to wait on some third-party API, or on writing or reading data to disk. You could speed the code in curl up by 100x and you probably wouldn’t even notice, because curl spends all of its time waiting on the network. Slow languages already delegate work to fast languages What about work that is obviously not I/O-bound? For instance, wikis have to turn stored markdown into renderable HTML3. Wouldn’t rewriting that in a faster language cause a massive speedup? It would! That’s why applications that do that use libraries like cmark-gfm that call out to fast languages for markdown rendering. In general, most slow languages can write very fast programs by invoking this kind of library. Python is famously a slow language, but it’s a competitive choice for CPU-bound workloads like machine learning and mathematics, because it has a healthy set of libraries that let it offload these workloads to fast C code. If a Python program spends 80% of its time …