Strengthening our defenses against distillation attacks
Anthropic has significantly reduced the effectiveness of AI model distillation attacks, which illicitly copy model capabilities, by implementing new defense methods that have cut harvested data by 90-99 percent.
Intelligence analysis by Gemini 2.5 Flash
Distillation attacks involve covertly extracting the capabilities of advanced AI models to replicate them in smaller, unauthorized models, often by foreign entities. Anthropic is concerned about the lack of robust safeguards in these distilled models and the national security risks they pose, leading them to develop and deploy new, more sophisticated defenses.
Imagine you have a super smart friend named Claude who's really good at solving puzzles, and you want to learn how they think. Distillation is like someone secretly watching Claude solve puzzles and then trying to teach a new, smaller friend to solve them the exact same way, without Claude's permission. Anthropic, the company that made Claude, has found new ways to make it much harder for people to secretly copy Claude's thinking, like making Claude only show the final answer, not all the steps, so the copycats can't learn as easily.
Analysis
Anthropic has recently announced substantial progress in mitigating distillation attacks, which are industrial-scale campaigns designed to extract and replicate the capabilities of advanced AI models without authorization. These attacks allow other entities, particularly overseas AI laboratories, to mimic frontier capabilities at a fraction of the usual development time and cost. The company's new methods have reportedly reduced the amount of data harvested by distillers by an impressive 90 to 99 percent, marking a significant win in the ongoing AI security arms race.
GLM-5.3
One of Anthropic's primary concerns regarding illicit distillation is the potential for these models to be deployed without adequate safeguards, leading to significant harm. The article highlights the case of Zhipu, which distilled the cyber capabilities of leading US models during the development of its GLM-5.3 model. Despite attempts to distill Claude Fable 5, Zhipu reportedly switched to an older Claude model and another lab's model after encountering Anthropic's improved cyber safeguards. The US Center for Advancing Innovation and Standards for Super Intelligence (CAISSI) has since identified GLM-5.3 as the most cyber-capable open-weights model released to date. Anthropic's own research indicates that GLM-5.3's safeguards are easily bypassed, and the model is capable of developing security exploits end-to-end.
Beyond safety, illicitly distilled models also present a national security risk. The capabilities gained through distillation are being integrated into foreign military, intelligence, and surveillance systems. An example cited is a case where requests to Moonshot’s Kimi models were secretly forwarded to Claude as part of distillation efforts. One such request, believed to be from an affiliate of China’s People’s Liberation Army, sought to analyze CCTV surveillance of a specific individual. This illustrates the type of sensitive capabilities that distillers aim to develop and exploit, underscoring the geopolitical implications of these attacks.
Claude Opus 4.8
Anthropic's defense strategy primarily focuses on making Claude's reasoning — the internal thought process before generating a response — harder for distillers to access. This reasoning is far more valuable than the final output alone, as it helps smaller models understand how Claude arrives at its answers. Initially, Anthropic implemented simple measures like summarizing Claude's reasoning before showing it to users. This significantly reduced the informativeness of transcripts for distillers, as demonstrated by a model trained on summarized transcripts of Claude Opus 4.8 achieving 71.8% on the SWE-bench Verified coding benchmark, compared to 80.4% for a model trained on full transcripts. However, distillers quickly adapted, moving from passive to active extraction methods, using sophisticated prompts to trick Claude into revealing its full reasoning. This necessitated the development of more advanced safeguards capable of detecting and blocking these persistent and inventive attacks, even at scale, involving trade-offs such as turning away significant revenue and requiring legitimate users to update their integrations.
90 to 99 percent
The success of Anthropic's strengthened defenses is evident in the reported reduction of harvested data by 90 to 99 percent. This substantial decrease indicates a significant blow to the effectiveness of distillation campaigns. The company's efforts involved substantial investment from its research and product teams and required turning away meaningful revenue and trillions of tokens' worth of paying traffic from distillers. While these measures have involved trade-offs, including some inconvenience for legitimate users who needed to update their integrations, Anthropic emphasizes that the costs are justified given the severe risks posed by illicit distillation. The company hopes that sharing its experiences will help other model developers and platforms better understand and implement their own defenses against these pervasive attacks, fostering a more secure and responsible AI development ecosystem across the industry.
Key points
- Anthropic has significantly reduced the effectiveness of AI model distillation attacks, cutting harvested data by 90-99 percent.
- Distillation attacks illicitly copy advanced model capabilities, often by foreign entities, posing risks to AI safety and national security.
- Examples include Zhipu's GLM-5.3 model, which distilled US cyber capabilities, and requests to Moonshot's Kimi models secretly forwarded to Claude for military surveillance analysis.
- Anthropic's defenses focus on making Claude's internal reasoning harder to access, moving from simple summarization to detecting active extraction attempts.
- The company has incurred substantial costs, including lost revenue and development effort, but deems these trade-offs necessary to combat the threats.
Anthropic's significant success in reducing data harvested by distillers by 90-99 percent demonstrates that effective defenses against illicit AI model copying are possible. This could lead to a more secure AI ecosystem where advanced capabilities are protected, fostering responsible development and preventing the proliferation of powerful models without robust safety measures.
Despite Anthropic's progress, distillers are persistent and inventive, suggesting an ongoing arms race where new methods of attack will likely emerge. The trade-offs, including turning away revenue and requiring legitimate users to update integrations, highlight the continuous cost and effort required to maintain these defenses, potentially impacting innovation and user experience.
