Detecting and countering malicious uses of Claude: March 2025

Anthropic's latest report sheds light on the increasingly sophisticated methods malicious actors are employing to exploit large language models (LLMs) like Claude. The company emphasizes its commitment to maintaining the utility of its models for legitimate users while simultaneously combating adversarial circumvention of safety measures. This ongoing cat-and-mouse game underscores the inherent challenges in deploying powerful AI systems responsibly, as threat actors continuously adapt their tactics.

Influence-as-a-service

The most significant and novel misuse identified by Anthropic involves a professional 'influence-as-a-service' operation. This operation leveraged Claude not merely for content generation, but as an orchestrator to decide when social media bot accounts should interact with authentic users through comments, likes, or re-shares. This represents a significant evolution in influence operations, moving beyond simple content creation to semi-autonomous, politically motivated engagement decisions, indicating a future where AI agents could manage complex disinformation campaigns with minimal human oversight.

This particular operation managed over 100 social media bot accounts across platforms like Twitter/X and Facebook. These accounts were assigned distinct political personas, engaging with tens of thousands of authentic social media accounts across multiple countries and languages. The sophistication of using Claude to make tactical engagement decisions based on client political objectives highlights the potential for AI to scale and automate malicious activities, making detection and attribution increasingly difficult for platforms and security researchers alike.

Clio

To combat these advanced threats, Anthropic's intelligence program acts as a crucial safety net, identifying harms not caught by standard scaled detection systems and providing context on how bad actors exploit their models. In investigating these cases, the team applied techniques described in their recently published research papers, including 'Clio' and hierarchical summarization. These methods are vital for efficiently analyzing the vast volumes of conversation data generated by LLMs.

Clio, alongside other classifiers, plays a critical role in analyzing user inputs for potentially harmful requests and evaluating Claude's responses before or after delivery. This multi-layered approach allows Anthropic to detect, investigate, and ultimately ban accounts associated with malicious activities. The continuous refinement of such intelligence programs and analytical tools is essential for staying ahead of threat actors who are constantly seeking new vulnerabilities and methods to bypass existing safeguards.

100 social media bot accounts

The report details how an actor utilized Claude to orchestrate over 100 social media bot accounts for a financially-motivated influence-as-a-service operation. These bots were used to push clients' political narratives, consistent with state-affiliated campaigns, though direct attribution was not confirmed. The ability of a single actor to manage such a large network with AI assistance demonstrates the force multiplier effect of generative AI for malicious purposes.

Beyond influence operations, Anthropic also observed other forms of misuse, including credential stuffing operations, recruitment fraud campaigns targeting job seekers in Eastern European countries, and a novice actor using AI to generate malware beyond their typical skill level. While successful deployment of these efforts was not confirmed in all cases, the potential for generative AI to accelerate capability development for less sophisticated actors is a significant concern. This trend suggests that AI could democratize access to advanced malicious tools, lowering the barrier to entry for cybercrime and other harmful activities.