Learned Relay Representations for Forward-Thinking Discrete Diffusion Models
A new method called Relay lets masked diffusion models carry useful internal information across decoding steps, improving speed and coding results.
Intelligence analysis by GPT-5.4 Mini
The paper argues that masked diffusion models throw away too much useful internal state between refinement rounds. Relay learns to pass that state forward, aiming to improve both quality and latency.
Some AI models make a guess, fix it a little, then guess again. The problem is that they forget helpful thinking from the last round, like erasing notes after every step.
This paper adds a tiny moving notebook inside the model. That notebook carries useful hints forward so the model does not have to start over each time.
The result, according to the paper, is faster thinking and better answers on some tasks like coding. It is a bit like a chef who keeps a list of what already works instead of tasting every ingredient from scratch each time.
Analysis
What the paper proposes
The paper focuses on masked diffusion models, which generate sequences by repeatedly refining masked tokens. The authors argue that each round of refinement discards rich internal computation, so the model has to rebuild useful information from scratch on the next pass.
To address that, they introduce Learned Relay Representations or Relay. The core idea is to add a differentiable per-token channel that carries latent information from one forward pass to the next. Rather than resetting between denoising rounds, the model is trained to propagate useful state forward, using truncated backpropagation through time.
What they claim it enables
The authors say Relay can be scaled to state-of-the-art diffusion language models and works alongside techniques such as block diffusion and KV caching. They first use a Sudoku-style planning task to justify the design choices, then scale the method to Fast-dLLM v2.
On the reported experiments, Relay outperforms standard supervised fine-tuning on coding tasks and reduces inference latency by up to 32%. The paper frames this as moving the performance-latency tradeoff forward, rather than forcing a choice between stronger outputs and slower decoding. The paper also says code for the experiments is available.
Key points
- The paper says masked diffusion models lose useful internal state between refinement rounds.
- Relay adds a learned per-token channel to carry latent information forward.
- The method is trained with truncated backpropagation through time.
- The authors report strong results on a Sudoku planning task and coding tasks.
- They claim up to 32% lower inference latency on Fast-dLLM v2.
