San Francisco Demands Apple and Google Delete AI ‘Nudify’ Apps From App Stores
San Francisco has issued cease-and-desist letters to Apple and Google, demanding the removal of 13 AI-powered 'nudify' apps from their app stores that generate nonconsensual intimate images.
Intelligence analysis by Gemini 2.5 Flash
San Francisco City Attorney David Chiu is taking legal action against Apple and Google, accusing them of 'aiding and abetting' the sale of explicit deepfake images through apps that allow users to create AI-generated nonconsensual nudes. The city demands the tech giants remove these apps, stop profiting from them, and improve moderation to prevent their appearance, citing California l…
Imagine there are apps that use special computer tricks, like magic, to make it look like people are naked in pictures, even when they're not. San Francisco is telling big companies like Apple and Google to take these apps out of their app stores because it's wrong and hurts people, especially kids, and the companies shouldn't make money from them.
Analysis
San Francisco's Legal Offensive Against AI Abuse
San Francisco City Attorney David Chiu has launched a significant legal challenge against tech giants Apple and Google, demanding the immediate removal of 13 AI-powered applications from their respective app stores. These apps are designed to 'nudify' or 'undress' individuals, creating nonconsensual intimate images through AI generation. The cease-and-desist letters, sent on Thursday, accuse Apple and Google of 'aiding and abetting' the sale of explicit deepfake images and profiting from this harmful technology, with Chiu estimating millions of dollars in fees collected by the companies. The city's legal stance is grounded in California laws that prohibit supporting services involved in creating deepfake pornography, emphasizing the platforms' responsibility to ensure their app stores do not facilitate sexual abuse.
The Pervasive Threat of AI Nudification
The emergence of deepfake 'nudification' technology has created a highly lucrative and deeply damaging online industry. These apps, websites, and bots enable users, predominantly men, to upload images of individuals, overwhelmingly women and girls, and digitally remove clothing or place them into graphic sexual scenarios with just a few clicks. The underlying generative AI technology has advanced rapidly, making these images and videos increasingly realistic and often available for free or a small fee. Research by organizations like the Tech Transparency Project (TTP) and universities has repeatedly identified hundreds of such apps on both Apple's App Store and Google's Play Store, with some even rated as suitable for children. The impact on victims is severe, leading to bullying, humiliation, reputational damage, mental health crises, and even suicidal ideation, as highlighted by Chiu.
Platform Accountability and Enforcement Challenges
Despite having developer policies that explicitly prohibit pornography, abuse, and harassment, Apple and Google have struggled to effectively police their platforms against these 'nudify' apps. Both companies have previously removed dozens of such applications following reports from researchers and journalists, with Google stating it has suspended 'hundreds' of violating apps and restricted related search terms. However, the persistent re-emergence of these apps, often disguised as 'face-swapping' tools, indicates a significant challenge in enforcement. Watchdog groups like TTP have expressed frustration, noting that despite repeated reports, the problem remains widespread, contradicting the companies' marketing claims of trusted and safe app stores. This ongoing struggle raises critical questions about the efficacy of current content moderation strategies and the need for more proactive and robust measures to combat AI-driven abuse.
Key points
- San Francisco has sent cease-and-desist letters to Apple and Google, demanding the removal of 13 AI 'nudify' apps from their app stores.
- City Attorney David Chiu accuses the tech giants of 'aiding and abetting' the sale of explicit deepfake images and profiting from them.
- The apps allow users to create nonconsensual intimate images, primarily targeting women and girls, with severe psychological impacts on victims.
- Both Apple and Google have policies against pornography and abuse but have struggled to consistently remove these apps despite repeated reports from researchers.
- The city's action highlights the ongoing challenge for tech platforms in moderating harmful AI-generated content and enforcing their own policies.
If Apple and Google take decisive action, it could significantly reduce the availability of these harmful apps, protecting potential victims and setting a precedent for greater platform responsibility in moderating AI-generated content. This could lead to more robust content moderation policies and better enforcement mechanisms across app stores.
Despite legal pressure, the cat-and-mouse game between app developers and platform moderators might continue, with new 'nudify' apps constantly emerging under different guises. The underlying technology is easily accessible, making complete eradication difficult, and the profit motive for developers remains strong.
