Introducing vLLM TorchTPU

Today we're releasing vLLM TorchTPU (vllm-torchtpu), a new backend for running vLLM on Google Cloud TPUs. It's built on Google's TorchTPU stack and runs the model definitions that already live in vLLM, so most models that work in vLLM today can be served on TPU without changes to the model code. This work comes out of the engineering partnership between Inferact and Google we announced last month. Our goals for the project are to: Make it easy to bring new models to TPU by reusing vLLM's existing model code. Push TPU inference performance in open source, with Pallas kernels for attention, Mixture-of-Experts, and quantized matmul. Keep the vLLM experience the same: the same CLI, OpenAI-compatible server, scheduler, and KV cache manager you use on other hardware. How It Works vllm-torchtpu is an out-of-tree vLLM platform plugin. Once installed, vLLM picks it up automatically through the vllm.platform_plugins entry point. Models run through vLLM's standard model runner and are compiled with torch.compile(backend="tpu"), which lowers to XLA. Each TPU device gets its own worker process, the same way vLLM runs on GPUs, and communication between devices goes through torch.distributed. This means the scheduler, workers, KV cache manager, and API server are all unmodified upstream vLLM. The TPU-specific work sits underneath them, mostly in a set of JAX/Pallas kernels that are registered as custom ops and called from inside the compiled graph. Getting Started Install pip install vllm-torchtpu This requires Python 3.12 on a Cloud TPU VM, and installs a matching vLLM, torch, torch_tpu, and libtpu alongside the plugin. The installation guide also covers uv, source builds, and Docker. Serve a model vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8
--tensor-parallel-size 8
--enable-expert-parallel
--kv-cache-dtype fp8
--max-model-len 16384
--async-scheduling The first startup compiles TPU graphs for the configured shapes, which can take a few minutes. Set VLLM_XLA_CACHE_PATH to reuse compiled artifacts across restarts (see Compilation & Warmup Cache). Speculative decoding, prefix caching, multi-LoRA, structured outputs, and multimodal inputs are all enabled with the usual vLLM flags. See the quickstart for more examples, including offline batch inference, multimodal models, and multi-LoRA. Kernels vllm-torchtpu uses custom Pallas kernels for complex operations that benefit from hand-tuning on TPU. Some examples: Ragged Paged Attention v3 for MHA/GQA …