Certification from Examples is Hard for Circuits and Transformers under Minimal Overparametrization
The paper argues that certifying learned models from examples can become exponentially hard, even with tiny extra capacity, for circuits and Transformers.
Intelligence analysis by GPT-5.4 Mini
This paper studies exact certification: how many labeled examples are needed to prove a learned hypothesis matches the target. It shows that for some circuits and log-precision Transformers, even minimal overparametrization can make that proof exponentially large.
This paper is about proving a machine is exactly right using examples. Sometimes that proof is easy. But the paper shows that a tiny change in the machine can make the proof blow up into something enormous.
It is a bit like trying to prove a lock is safe by checking keys. If the lock gets only slightly more complicated, the number of keys needed to test can become huge.
The main warning is that a model can seem fine on normal tests and still be very hard to fully check. That matters when the machine is supposed to do careful thinking or math.
Analysis
What the paper asks
The paper looks at exact certification from examples: given a learned hypothesis, how many labeled examples are needed to prove that it is exactly the target function? That matters because a model can look good on average and still behave inconsistently on specific inputs.
Main result
The authors show that certification can become exponentially hard even when the model is only minimally overparameterized. For threshold circuits of depth at least 2, adding just one extra gate can force certificate sizes to grow exponentially with the input dimension. They prove a similar hardness result for log-precision Transformers, with only constant architectural overhead.
Approximate certification
The paper also studies weaker forms of certification. Allowing a polynomial number of mistakes still does not avoid the explosion in certificate size. By contrast, constant relative-error guarantees can hide exponentially many mistakes, which means a coarse certificate may miss a very large number of failures.
Empirical section
The authors test constructed circuits and trained Transformers on binary addition. The constructed circuits match the theory and show the exponential barrier directly. For trained Transformers, the paper reports that imperfect models can slip past large certificate candidates drawn uniformly at random.
Takeaway
The paper’s core message is that small amounts of extra capacity can make exact verification from examples dramatically harder. For AI systems used in reasoning or algorithmic tasks, that is a warning that validation pipelines based only on sampled examples may be too weak to prove correctness.
Key points
- Exact certification asks for the smallest labeled example set that proves a model matches the target.
- For depth-2-or-more threshold circuits, one extra gate can make certificates exponentially large.
- A similar hardness result holds for log-precision Transformers with constant overhead.
- Approximate certification can still require exponentially large certificates unless the error tolerance is very coarse.
