Training only the "Super Weights" of an LLM fails — importance does not imply trainability (COLM 2026)

This work (Super Weights) shows experimentally why the seemingly sensible idea of training the important parameters does not work.

1What Super Weights are

Recent work identified Super Weights: individual parameters whose removal degrades model performance by orders of magnitude. This paper first shows the pruning-induced degradation does not universally apply to all LLMs. It then poses the natural question — if these parameters matter so much, Super Weight-aware training should be effective — and shows the opposite is true.

2Training them in isolation loses accuracy

Training Super Weights in isolation (100 to 8,192 parameters) drops accuracy to random-guessing levels on both OLMo-1B and OLMo-7B. Expanding the target to local neighborhoods of up to 36K parameters yields no improvement. Crucially, the failure is specific to Super Weight coordinates: training an equal number of randomly chosen positions in the same down_proj layers instead improves over baseline.

So the collapse comes not from training sparsely (few parameters) but from targeting Super Weights themselves.

3Vanilla LoRA succeeds with 0.16% of parameters

By contrast, vanilla LoRA — updating every position in attention weight matrices through low-rank structure — succeeds with only 0.16% of parameters, and the same low-rank update on down_proj succeeds too. A 10-seed ablation confirms that constraining LoRA updates to positions corresponding to Super Weight coordinates yields statistically indistinguishable results.