Supervised Fine-Tuning vs. In-Context Learning: An Equilibrium Analysis of LLM Personalization under Congestion
A study on the trade-offs between Supervised Fine-Tuning and In-Context Learning for Large Language Models, focusing on the impact of congestion on user incentives and platform design.
Intelligence analysis by Llama
The paper explores the equilibrium analysis of LLM personalization under congestion, highlighting the interplay between pretraining coverage and data signal-to-noise ratios, and the non-monotonicity of resource consumption.
Imagine you have a super-smart AI assistant that can answer questions and do tasks for you. But, it's not perfect and needs to learn from you. There are two ways to make it better: one is to give it a lot of data to learn from, and the other is to teach it specifically for a task. But, what if many people want to use the same AI assistant at the same time? It's like a traffic jam, and the AI assistant gets congested. This paper studies how to make the AI assistant better and how to manage the congestion so that everyone can use it efficiently.
Analysis
A Critical Tension in LLM Serving
Large Language Models (LLMs) have revolutionized AI services, but a critical tension emerges: while personalization improves model performance, it consumes scarce computational resources that users must share. When should a user invest in expensive Supervised Fine-Tuning (SFT) versus lightweight In-Context Learning (ICL)? How does congestion from other users' personalization choices reshape these incentives? And what strategies should platforms adopt when offering multiple personalization algorithms?
Our Framework and Insights
We develop a tractable framework for LLM serving that captures the statistical-economic trade-offs users face. Our analysis yields several surprising insights. First, we show that ICL and SFT dominate in different regimes, determined by an interplay between pretraining coverage and data signal-to-noise ratios, but congestion can flip these rankings. Second, equilibrium resource consumption exhibits pronounced non-monotonicity: improving pretraining precision reduces the congestion, while broader pretraining coverage and harder tasks sometimes increase it. Third, we prove that offering both personalization methods never hurts the platform's maximal profits, despite potentially increasing computational load.
Experimental Validation and Platform Implications
Experiments with GPT-2 on linear regression tasks validate our theoretical predictions about algorithm performance. Complementing these results, our review of documentation from 21 major AI platforms shows that the share offering both SFT and ICL increased from 9.5% in 2021 to 71.4% in 2025, consistent with our platform-design implications.
Key points
- Supervised Fine-Tuning (SFT) and In-Context Learning (ICL) dominate in different regimes, determined by pretraining coverage and data signal-to-noise ratios.
- Congestion can flip the rankings of SFT and ICL.
- Equilibrium resource consumption exhibits non-monotonicity.
- Offering both SFT and ICL never hurts the platform's maximal profits.
If platforms adopt our suggested strategies for offering multiple personalization algorithms, it could lead to more efficient use of computational resources and improved model performance, ultimately benefiting users and the AI industry as a whole.
However, the increased complexity of offering multiple personalization methods could lead to higher development and maintenance costs for platforms, potentially offsetting the benefits of improved model performance and resource efficiency.
