Mechanistic origins of catastrophic forgetting: why RL preserves circuits better than SFT?
A paper finds RL preserves more of a model's base circuits than SFT, but SFT adapts faster to the new task.
Intelligence analysis by GPT-5.4 Mini
The paper compares reinforcement learning and supervised fine-tuning on Qwen2.5-3B-Instruct and looks at forgetting at the level of internal circuits. It finds a trade-off: SFT learns the target task faster but disrupts more of the original model, while RL keeps more of the base circuitry intact.
A big language model is like a machine with many tiny parts working together. When it learns a new job, some learning methods change those parts a lot, and old skills can get damaged.
This paper says two ways of teaching the model behave differently. One way, called SFT, helps it learn the new job faster, but it scrambles more of the old parts. The other way, RL, keeps more of the old parts working, but learns the new job more slowly.
It is like repainting a toy car. One method changes the whole car quickly, while the other keeps more of the original shape. The paper says that may be why RL forgets less than SFT.
Analysis
What the paper asks
Fine-tuning large language models often causes catastrophic forgetting, where new training weakens old abilities. The paper asks whether the common observation that reinforcement learning (RL) forgets less than supervised fine-tuning (SFT) also shows up inside the model, not just in behavior.
What the authors measured
The authors introduce differential circuit vulnerability, described as a head-level measure of how much a circuit degrades during fine-tuning. They use it to compare RL and SFT on Qwen2.5-3B-Instruct adapted to scientific question-answering.
Main result
The paper reports a clear trade-off. SFT adapts more quickly to the target task, but it also causes substantially greater circuit disruption and more forgetting of prior capabilities. RL preserves a larger fraction of the base circuit, but it does so with slower task adaptation.
Why that matters
The authors argue that this circuit-level preservation may help explain why RL is more resistant to catastrophic forgetting than SFT. In other words, the difference is not only about outputs or scores; it may reflect how much of the model's original internal machinery survives the update. The paper positions this as a mechanistic extension of earlier behavioral findings about RL staying closer to the base policy.
Key points
- The paper studies why fine-tuning can cause catastrophic forgetting in language models.
- It compares reinforcement learning and supervised fine-tuning on Qwen2.5-3B-Instruct.
- The authors introduce differential circuit vulnerability as a head-level measure of circuit damage.
- SFT learns the new task faster but disrupts more of the original model.
- RL preserves more of the base circuit but adapts more slowly.
