Day-0 Support for Flint on vLLM

vLLM supports Recursive's Flint from day 0. Both its NVFP4 and BF16 checkpoints run on a single GPU, and a single request decodes at about 500 tokens/s on one GB200, even with 100K tokens of context. Flint is a 40B-parameter MoE that activates only about 2.5B parameters per token, with sliding-window attention in most layers and a 262K-token context. It ships as two separate checkpoints: NVFP4 (quantization-aware trained) and BF16. Quick start The day-0 Docker image vllm/vllm-openai:flint (x86_64 and arm64, CUDA 13.0) has Flint support built in:

NVFP4 checkpoint (GGUF): config and tokenizer come from the BF16 repo

docker run --gpus all --ipc=host -p 8000:8000
-v ~/.cache/huggingface:/root/.cache/huggingface
vllm/vllm-openai:flint
RecursiveSI/flint-nvfp4-gguf
--hf-config-path RecursiveSI/flint --tokenizer RecursiveSI/flint
--enable-auto-tool-choice --tool-call-parser glm47
--reasoning-parser glm47 --exclude-tools-when-tool-choice-none

BF16 checkpoint

docker run --gpus all --ipc=host -p 8000:8000
-v ~/.cache/huggingface:/root/.cache/huggingface
vllm/vllm-openai:flint
RecursiveSI/flint
--enable-auto-tool-choice --tool-call-parser glm47
--reasoning-parser glm47 --exclude-tools-when-tool-choice-none Flint support is being merged into vLLM (#62938). With a vLLM build that includes it, serve the same way:

NVFP4 checkpoint (GGUF): config and tokenizer come from the BF16 repo

vllm serve RecursiveSI/flint-nvfp4-gguf
--hf-config-path RecursiveSI/flint --tokenizer RecursiveSI/flint
--enable-auto-tool-choice --tool-call-parser glm47
--reasoning-parser glm47 --exclude-tools-when-tool-choice-none

BF16 checkpoint

vllm serve RecursiveSI/flint
--enable-auto-tool-choice --tool-call-parser glm47
--reasoning-parser glm47 --exclude-tools-when-tool-choice-none Model architecture Flint is a 49-layer pre-norm MoE with hidden size 2048. What sets it apart is how little of it looks at the whole context. Figure 1: Flint's layer stack and its two sublayers. Gold marks what differs from a standard pre-norm MoE. Open it full-screen. Mostly local attention. 35 layers attend to a 513-token sliding window with RoPE; the other 14 attend to the full context with no positional encoding. Only those 14 keep KV for the whole sequence, so the cache costs 28 KiB per token. Learned attention sinks, GQA with 32 query and 4 KV heads, and per-head Q/K RMSNorm. 256 experts, sigmoid top-6, plus a shared expert. Layers 0 and 1 use a dense MLP …