OpenAI, Google, Microsoft and Anthropic back one robots.txt signal to opt out of AI training
OpenAI, Google, Microsoft and Anthropic said they will honor a shared robots.txt signal, drafted in the IETF's AI Preferences working group, that lets a website state one AI-training preference for every participating provider instead of blocking each company's crawler separately.
What happened
- On October 30, 2026, OpenAI said it will honor a common robots.txt signal for foundational generative AI training and aims to implement it in the coming months; its existing crawler controls stay in place.
- Google said the same day it plans to support the shared 'policy token' in 2027, and will treat an opt-out as if the site had disallowed Google-Extended unless Google-Extended is set explicitly.
- The Verge reported that Anthropic, whose standards lead proposed the signal in the IETF, has already adopted it, that Microsoft and Google plan to support it next year, and that Bing search results won't be affected by a site's choice.
- The signal is voluntary and builds on the IETF AI Preferences (AIPREF) working group's draft; company-specific controls such as Google-Extended keep working alongside it.
- Meta signaled support for a common signal but did not say whether it will honor it, according to The Verge.
Why it matters: Small business sites and publishers can soon say no to AI training with one rule instead of tracking every AI crawler by name. It only covers participating companies, it's a preference rather than a block, and Google notes that allowing Google-Extended is what keeps a site eligible to be cited in Gemini answers.
Sources
- OpenAI: Making it easier for websites to express AI training preferences openai.com
- Google Search Central Blog: Supporting a shared AI training control for websites developers.google.com
- The Verge: The biggest names in AI are backing a new way to opt out of scraping theverge.com
Checked October 31, 2026 against the sources above. Claims about performance or benchmarks are the company's own unless stated otherwise.