Testing the Test: Score-Direction Instability in Class-Split Anomaly Detection
A paper shows class-split anomaly detection can break when the held-out class overlaps the normal data in representation space.
Intelligence analysis by GPT-5.4 Mini
The authors argue that a common within-dataset anomaly benchmark can become ill-posed when the anomaly class is too close to the normal mixture. In that regime, scores can drift toward random chance or even flip direction, and the best direction may depend on which hidden class is held out.
The paper says an anomaly test can be unfair if the hidden class looks too much like the normal data. It is like trying to spot a lost red ball in a pile of red toys: sometimes the test stops telling who is really different.
Analysis
What the paper claims
The paper examines within-dataset class-split evaluation, a common proxy for fully unconditional out-of-distribution anomaly detection. In this setup, one class is held out as the anomaly class and the rest are treated as normal. The authors argue that this protocol can become ill-posed when the held-out class overlaps the normal mixture in representation space.
What happens in the bad regime
According to the abstract, anomaly scores may collapse toward chance or even invert. That means a detector can appear to work poorly, or the preferred score direction can reverse, simply because the chosen anomaly class sits too close to the normal data in feature space. The paper says this direction choice can depend on the unknown anomaly class itself, which makes the benchmark unstable as a measure of unconditional anomaly detection skill.
Their diagnostic
The paper introduces a training-free diagnostic called neighborhood class leakage. The authors report that it predicts score-direction instability across Fashion-MNIST, CIFAR-10, and Imagenette. They test this in both pixel space and VAE latent space, suggesting the effect is not tied to one particular representation.
Implication for evaluation
The main takeaway is not that class-split anomaly detection is useless, but that it is geometry-dependent. The authors suggest these benchmarks should be treated as stress tests of representation geometry rather than as unconditional evidence that a method can detect anomalies in general. That is a narrower, more careful standard for interpreting results.
Key points
- The paper studies class-split anomaly detection as a proxy for out-of-distribution detection.
- It argues the protocol can become ill-posed when the held-out class overlaps the normal mixture in representation space.
- In that regime, anomaly scores may collapse toward chance or even invert.
- The preferred score direction may depend on the unknown anomaly class.
- A training-free diagnostic called neighborhood class leakage is reported to predict this instability.
If the diagnostic works broadly, it could help researchers spot when anomaly benchmarks are misleading before they draw conclusions. That could lead to fairer comparisons and better-designed tests for anomaly detection systems.
If benchmark instability is common, some reported gains in anomaly detection may be partly an artifact of the chosen class split. That would make it harder to know whether a method truly generalizes beyond the dataset geometry.
