Training RNNs without recurrence — "Supervised Memory Training (SMT)," parallelizable across time

This work (SMT) trains recurrent neural networks (RNNs) without using recurrent credit propagation.

Training RNNs requires assigning credit across long sequences of computation. Standard backpropagation through time (BPTT) handles this poorly: it is sequential in time, limiting parallelism, and suffers from vanishing or exploding gradients, making long-range associations hard to learn.

The proposed Supervised Memory Training (SMT) sidesteps recurrent credit propagation entirely by reducing RNN training to supervised learning on one-step memory-transition labels (m_t, x_{t+1}) → m_{t+1}. These memory labels are acquired by training a Transformer-based encoder on a predictive-state objective — retaining only information from the past necessary to predict the future. By decoupling what to remember from how to update memory, SMT enables time-parallel RNN training with a stable O(1)-length gradient path between any two tokens, without ever unrolling the RNN.

The authors find that SMT outperforms BPTT when pretraining various RNN architectures on tasks like language modeling and pixel-sequence modeling. SMT enables nonlinear RNNs to better capture long-range dependencies and train in parallel, potentially unlocking the scaling of models that build temporal abstractions of past experience.