Private Claude Chats Exposed in Google and Bing Search Results

A Collision of Functions: Web Crawlers and AI Chatbots

The recent exposure of private chats on the AI chatbot Claude in Google and Bing search results has raised questions about the responsibility of companies like Anthropic and Google in ensuring that sensitive information is not exposed. The issue appears to have been caused by a lack of proper indexing, which highlights the importance of understanding how web crawlers and AI chatbots interact.

Anthropic instructs web crawlers, like those used by search engines like Google and Bing, not to index chats a user decides to share with other people. The company does this via something called a robots.txt file, which has long been considered the standard way to tell web scrapers what parts of a site are appropriate to access. However, it's harder to prevent pages from being included in search engine results than you might expect.

Bing, which still shows "about 612 results" if you search "site: claude.ai/share" at the time of this writing, says in its technical documentation that developers can block the search engine's web crawlers using robots.txt—but that web developers should also include a "noindex" tag on individual pages as well. Some of the chats that showed up in search results flagged in the Reddit post were deleted by the time WIRED viewed them, and results no longer show up in Google when you search the "share" query that still worked on Bing.

In a developer guide, Google says that it ignores robots.txt instructions if that page is linked to from elsewhere on the internet and the page owner doesn’t also include a special "noindex" html tag on the page or a "x-robots-tag" in the page’s response header. WIRED reviewed a sample of the exposed Claude chat pages and found that they did not include the "noindex" tag that both Bing and Google say they take into consideration when deciding whether or not to index a page.

Microsoft, which owns Bing, did not provide comment ahead of publication. Anthropic did not respond to multiple requests for comment. Google spokesperson Ned Adriance tells WIRED that the indexing of shared Claude chats is Anthropic’s responsibility. "Neither Google nor any other search engine controls what pages are made public on the web, and these pages were indexed across many search engines," Adriance says. "We give site owners clear controls to decide whether pages can be crawled or indexed, and we always respect those directives."

Anthropic didn’t respond to questions about why it didn’t include the "noindex" tag on the shared chat pages. Last September, Anthropic got heat for the same issue, and told Forbes that it uses robots.txt to let crawlers know they shouldn’t access the shared chats. But as the Forbes report points out, there’s no guarantee that will stop search engines from indexing specific web pages.

Even if robots.txt doesn’t always work to prevent pages from being indexed on search engines, AI labs are still making use of it for other purposes. Many labs promise creators that their websites won’t be used as AI training fodder so long as the developers make sure to "disallow" certain crawlers in their robots.txt files, and the labs themselves take that advice as well. Anthropic, Meta, and OpenAI all include instructions in their chatbots’ robots.txt files that "disallow" their competitors’ web crawlers from accessing any part of the website where the chatbots are hosted.

OpenAI and Meta did not respond to a request for comment about the practice. Google did not address WIRED’s questions about the fact that its competitors are blocking its AI training crawler, Google-Extended, from their chatbots’ sites. Though shared chats are no longer turning up in Google results, the pages WIRED reviewed still don’t have a "noindex" tag, meaning that they could potentially show up in search engines again.

Claude users can go to Settings > Privacy > Shared chats to manage access—and keep their secret conversations private.