Bayes-Sufficient Representations in Supervised Learning
The paper defines when a representation keeps exactly the information needed for a supervised task, with the target depending on both the loss and data distribution.
Intelligence analysis by GPT-5.4 Mini
The paper argues that “relevant information” in representation learning should be defined relative to a fixed prediction problem. It introduces Bayes-sufficient representations, Bayes quotients, and Bayes-minimality to describe how much information a model needs for Bayes-optimal prediction.
The paper says a model should only keep the clues it really needs for a job, like a backpack packing only the tools for one hike. What counts as “needed” changes with the task, so the same picture can be useful in different ways depending on what the model must predict.
Analysis
Core idea
The paper asks a simple but important question: for a given supervised task, what information in the input actually matters? Its answer is that the useful information depends on both the data distribution and the loss function. A representation is Bayes-sufficient if some prediction head can use it to produce a Bayes-optimal action for that problem.
Bayes quotient and minimality
The framework becomes more precise in the case where the Bayes action is almost surely unique. In that setting, inputs can be grouped into a Bayes quotient: two inputs belong together if they require the same Bayes-optimal action. A representation is sufficient when it is detailed enough to refine this quotient. It is Bayes-minimal when it contains no more information than the quotient itself.
Loss functions change what counts as relevant
The abstract highlights that the relevant target changes with the objective. Under zero-one loss, the Bayes class is what matters. Under squared loss, the conditional mean matters. For Brier loss in binary prediction, the conditional probability matters. For log loss and other strictly proper scoring rules, the predictive distribution matters. The main point is that the “right” representation is not universal; it is tied to the supervised problem.
Evidence from experiments
The paper reports controlled finite experiments, neural bottleneck experiments, and an iNaturalist taxonomic refinement experiment. These are used to show the difference between being sufficient for prediction, being minimal, and retaining extra information that is not required for the task. The abstract does not claim that the framework solves all representation-learning questions, but it does present a formal way to separate necessary from unnecessary information for a fixed supervised setting.
Key points
- The paper defines Bayes-sufficient representations for supervised learning tasks.
- What counts as relevant information depends on both the loss function and the data distribution.
- The Bayes quotient groups inputs that need the same Bayes-optimal action.
- Bayes-minimal representations contain no more information than that quotient requires.
- Experiments on neural bottlenecks and iNaturalist illustrate sufficiency versus extra retained information.
If the framework holds up, it could help researchers build representations that are smaller and better matched to a task. It may also make it easier to compare bottlenecks and understand when extra features are truly unnecessary.
The main risk is that the framework is only as useful as the task definition and loss function behind it. In practice, real systems may still keep extra information for robustness, transfer, or downstream uses that the Bayes-optimal view does not capture.
