All news

Ant Group and vLLM open-sourced Humming, a quantized inference kernel library that covers NVIDIA GPUs from Turing on and other CUDA-compatible chips.

Tests which LLM inference optimizations designed for NVIDIA GPUs actually help on Apple Silicon.

FlashMap uses flow maps to improve the speed and acceptance trade-off in speculative decoding.

Specializing the tokenizer vocabulary of a fine-tuned small model cut token counts and inference cost by 77% in a production essay grader.

Re-selects KV cache visual tokens during long multimodal generation based on drift, under a budget.

Cerebras explains how it deploys trillion-parameter models across wafer-scale chips that each hold only 44 GB of on-chip SRAM.

Mix-or-Split schedules LLM inference across devices, edge and cloud using a fine-grained latency model.

PrefixQuant adds prefixed tokens to absorb activation outliers, enabling simpler static LLM quantization.

Fine-tuning Gemma 4 26B-A4B to think 39 to 69 percent less with the same accuracy.

PyTorch 2.15 brings FP8 and faster decoding kernels to the Apple MPS backend.

Uses contextual sparsity to get around the memory bandwidth limit in edge LLM inference.

EcoxPU orchestrates disaggregated LLM inference across heterogeneous accelerators with agents to cut energy use.

DWI speeds up on-device LLM decoding by picking the width of a nested model per step.