Removing the scaffolding built to prevent collapse — simplifying video pretraining and cutting its compute several-fold
Self-supervised learning from video has leaned on elaborate scaffolding to stop representations from collapsing to a single answer. This work uses an objective that provably excludes collapse, removes that scaffolding, and reduces the architecture to an encoder and a projector, reporting parity with existing methods at a fraction of the pretraining compute.
Paper overview (our summary)
- Field (arXiv category)cs.CV（+1）
- AuthorsLukas Kuhn, Lucas Maes, Giuseppe Serra, et al. (7)
- Submitted2026-08-27
- arXiv ID2608.27395v1
Key points
- Self-supervised learning from video has accumulated architectural devices to prevent collapse, where the model returns the same representation regardless of input.
- The proposal uses an objective that provably excludes collapse, dispensing with the target encoder, stop-gradient and capacity-limited predictor alike.
- What is left is an encoder, a projector, and one knob to set.
- Since what the encoder pays for is how many tokens it looks at, discarding tokens at random shrinks the bill and, as reported, lifts downstream accuracy at the same time.
- At matched epochs on identical data it matches or surpasses a strong video baseline at 5.6 to 20.8 times less pretraining compute.
1Scaffolding put up to stop things collapsing
Learning representations without labels carries one failure that has to be avoided: collapse, where the model returns the same answer whatever it is shown. Once nothing needs to be distinguished, the loss falls and the result is useless.
To prevent this, methods have accumulated devices: keep a separate target encoder updated slowly, stop the gradient partway, deliberately limit the capacity of the predictor, or else reconstruct the pixels themselves. Every one of these is scaffolding against collapse rather than the thing anyone wanted to learn.
2What remains when the scaffolding comes down
Take the scaffolding down and there is less left to tune. What stands out here is less the performance than the simplification. The work asks whether the accumulated devices were ever necessary, and answers that they were not. Where the preceding articles described ways of avoiding rebuilding, this one goes back to the building side and removes assumptions from how the building is done.
3Seeing less turns out to be better
If cost is governed by how many tokens the encoder observes, dropping tokens uniformly makes it cheaper. What is interesting is the report that dropping them also improved downstream accuracy. Showing everything is not necessarily best.
For context, 8.0% of the AI papers held here concern pretraining from scratch while 18.9% concern transferring, adapting or fine-tuning something that already exists, so the center of gravity clearly sits with the latter.
4There is still room on the building side
Put in one sentence, the claim is that pretraining from video was thought to be expensive, and part of that expense was never intrinsic: it was the scaffolding put up to prevent collapse. Lower the compute barrier and using video as a general visual substrate becomes a realistic option. Even in a field where reuse has become the mainstream move, room remains to question how the thing is built in the first place.
Why it matters
Instead of adding components to raise performance, there is a direction that asks whether the accumulated components were needed at all. Take the scaffolding down and both the tuning surface and the cost shrink. Even in a field where reuse has become the dominant move, removing assumptions from how something is built can open a large amount of room.
FAQ
What is collapse?
Why would dropping tokens improve accuracy?
Is this about reusing an existing model?
Sources (primary)
Source: arXiv (descriptive metadata is CC0 public domain). Summaries are our own; see arXiv for the original text and PDF.
- arXiv abstract page (original, official)
- PDF (arXiv)
- arXiv ID: 2608.27395