PrefixQuant: Eliminating Outliers by Prefixed Tokens for Large Language Models Quantization

What happened

PrefixQuant (Mengzhao Chen et al., HKU and Shanghai AI Lab; now in IEEE TPAMI) makes 4-bit quantization of LLM weights, activations and KV cache much more accurate with a simple, training-free trick: put the tokens that cause outliers at the start of the KV cache. The IEEE version is paywalled, so this is based on the public arXiv version (2410.05265).

Results in the W4A4KV4 setting (4-bit weights, activations and KV) on Llama-3-8B:

• Average zero-shot accuracy over five reasoning tasks +3.08 points above SpinQuant with dynamic quantization, +2.85 with static quantization.
• Up to 2.74x faster prefill and 2.16x faster decoding with W4A4.
• The prefix step takes about one minute even for Llama-3-70B.

Why it works

Quantization maps values onto a few levels using a scale. One huge value (an outlier) forces a big scale, crushing every normal value into the same few levels. Earlier methods (SmoothQuant, rotations like QuaRot and SpinQuant) mostly attack channel-wise outliers: certain hidden dimensions that are always large.

But LLMs also have token-wise outliers: a few specific tokens (often the first token, or tokens like . and \n) act as attention sinks and carry massive activations. Because they can appear anywhere, they ruin per-tensor scales, especially for static quantization where scales are fixed ahead of time.

PrefixQuant's idea:

1. Find which tokens produce these outliers (offline, cheap).
2. Prepend them once and keep their KV in the cache in full precision.
3. The model now "sinks" attention onto that prefix, so the same outliers stop appearing in the real tokens.
4. Real tokens have a tame range, so static per-tensor scales work, which is faster than computing scales per token at runtime.
5. Optional block-wise fine-tuning recovers a bit more accuracy.

The bigger pattern

Isolate the outliers instead of fitting everything around them. It's the same principle as keeping hot keys out of a shared cache partition, or handling the long tail separately in a histogram. Understanding why outliers exist (attention sinks) leads to a much simpler fix than brute-force math.

What it means in practice

• 4-bit activations plus 4-bit KV is where big serving savings are (memory and bandwidth). Methods like this make it usable without a big quality drop.
• When evaluating a quantized model, ask which quantization (static vs dynamic, what handles outliers), not just "4-bit".
• General lesson for data systems: when a few items blow up your scale or capacity planning, special-case them.