Reading Calibrated Uncertainty from Language Model Trajectories
The paper reads uncertainty from layer-by-layer language model trajectories and says a sparse probe beats maximum softmax probability on abstention.
Intelligence analysis by GPT-5.4 Mini
Instead of treating a model's final confidence score as the whole story, the paper looks at how hidden states evolve across depth. It turns those trajectories into geometric features and uses a sparse probe to detect when uncertainty is building, reversing, or staying stable.
A language model is like a student solving a problem step by step. Most people only look at the final answer and the confidence number next to it.
This paper watches the whole path the student takes inside the model. It turns those steps into simple shape measurements and uses them to guess when the model is really unsure.
That matters because two students can end at the same answer but get there in very different ways. The path can show trouble before the final answer does.
Analysis
What the paper is doing
The paper argues that maximum softmax probability is a cheap baseline for uncertainty, but often a poorly calibrated one for language model generation with structured output. It also argues that probing a model's activations as if they were static snapshots misses something important: the path a representation takes through the layers.
Main idea
Rather than only inspect the endpoint, the authors trace the cumulative per-layer MLP updates and summarize that trajectory with eleven scale-invariant geometric features. Those features are then fed into a sparse linear probe. Because the features have closed-form geometric meanings, the probe is designed to stay interpretable instead of behaving like a black box classifier.
Findings
The abstract says the probe outperforms MSP under selective abstention. It also reports gains that grow as the baseline becomes more miscalibrated, reaching up to 21 AURC points. That suggests the trajectory contains useful uncertainty signals that final probabilities can hide.
Why the framing matters
The paper's interpretability claim is not just that the probe works, but that its coefficients can reveal where errors start to form: which layers commit too early, which layers push back against the current state, and where the trajectory drifts away from the final endpoint. In short, it treats uncertainty as something that unfolds across depth, not just something that appears at the end.
Key points
- The paper says final confidence scores are often miscalibrated for language model uncertainty.
- It measures the layer-by-layer trajectory of per-layer MLP updates instead of only the endpoint.
- Eleven scale-invariant geometric features feed a sparse linear probe with closed-form meaning.
- The probe beats maximum softmax probability under selective abstention, with gains up to 21 AURC points.
- The coefficients are meant to show where along depth errors begin to form.
