ERRORQUAKE: Heavy-Tailed Error Severity Distributions in Open-Weight Large Language Models
A paper says open-weight LLMs can share the same accuracy while differing sharply in how severe their mistakes are. It proposes scoring error severity, not just counting errors.
Intelligence analysis by GPT-5.4 Mini
The paper argues that error rate alone hides important differences between models. Using a new 10,000-query benchmark across 21 open-weight LLMs, it finds that severity distributions vary meaningfully even when accuracy is matched, and that this extra signal can distinguish models that look similar on standard scores.
A model can miss the same number of questions as another model, but one might make small slips while the other makes scary made-up answers. This paper says people should count not just how many mistakes happen, but how big those mistakes are.
Analysis
What the paper claims
The abstract argues that a single error count compresses too much information. Two models can land at the same accuracy while producing very different kinds of mistakes, and those differences are not visible if all errors are treated as equal.
The benchmark and measurements
To study that, the paper introduces Errorquake-10k, a 10,000-query benchmark that scores responses on a continuous 0-4 severity scale across 8 domains and 5 difficulty tiers. It then fits severity distributions for 21 open-weight models and estimates a severity distribution index, described as the upper-tail slope of a Gutenberg-Richter-style fit, with bootstrap confidence intervals.
Main findings
The abstract says 85 of 210 model pairs have disjoint 95% confidence intervals for this severity index even when their accuracy is close, defined as |Delta epsilon| < 0.05 on human-consensus scoring. It also reports a 519-item, three-rater human validation study with strong agreement, plus a high correlation between the LLM-judge ranking and human judgments. The paper further claims a non-reducibility theorem showing that severity profile and error rate are not redundant, since severity still carries information after conditioning on accuracy.
Why the result matters
The paper’s broader point is practical: severity can separate models that look similar under standard benchmarks. It also reports that error type changes with severity, with low-severity errors leaning toward retrieval problems and high-severity errors leaning toward fabrications. That supports the paper’s conclusion that severity distribution should be reported alongside accuracy.
Key points
- The paper says error rate alone hides meaningful differences in how models fail.
- It introduces Errorquake-10k, a 10,000-query benchmark with severity scoring from 0 to 4.
- Across 21 open-weight models, severity distributions differ even when accuracy is matched.
- The abstract says human validation supports the measurement approach and the judge ranking.
- The paper concludes that severity should be reported alongside accuracy.
If this approach is adopted, model evaluations could become more informative than a single accuracy score. That could help researchers and buyers spot systems that are safer or more reliable in the ways that matter most.
Severity scoring may be harder to standardize than simple error counts, especially across domains and judges. If benchmarks do not generalize well, the new metric could add complexity without fully replacing the simplicity of accuracy.
