Speeding up long-context LLMs by indexing once — "CLSA" cross-layer sparse attention

This work (CLSA) speeds up inference for LLMs handling long contexts.

Long-context inference in modern LLMs is increasingly constrained by decoding efficiency, especially in reasoning-heavy settings where models generate long intermediate chains of thought. Existing sparse-attention methods face a practical efficiency-quality trade-off: structured block-sparse methods accelerate strongly but lose noticeable quality, while token-sparse methods are usually more accurate yet deliver limited end-to-end speedup because top-k routing over the full cache remains expensive.

The authors propose cross-layer sparse attention (CLSA), built on top of KV-sharing architectures such as YOCO. The core idea is to share not only the KV cache across cross-decoder layers, but also the routing index. A single indexer computes token-level top-k selection once and reuses the resulting index across layers, preserving the fine-grained selectivity of token-sparse attention while amortizing the routing overhead. The resulting architecture improves all major inference bottlenecks jointly — pre-filling, KV-cache storage, and long-context decoding.

Across short- and long-context benchmarks, CLSA is both accurate and efficient, achieving up to 7.6x decoding speedup and 17.1x overall throughput improvement at 128K context — a more complete architectural solution for long-context LLMs that jointly advances model quality and inference efficiency.