OpenAI, Google and Microsoft say they'll honor one shared robots.txt line that lets websites opt out of AI training.

Website owners who don't want their pages used to train AI models currently have to block each company's crawler by name in robots.txt, the file that tells bots what they can and can't do on a site. On Friday, OpenAI, Google and Microsoft said they will honor a single shared signal instead (OpenAI, Google Search Central, Microsoft Bing). It comes from an IETF draft written by an Anthropic engineer, and Search Engine Land reports that Anthropic is part of the effort too. The signal is a special entry, User-Agent: __usage_ai-training, followed by Disallow: / to opt the whole site out, or by specific paths. It covers training generative AI models only. It doesn't block crawling or stop a page from appearing in search, and the old company-specific rules keep working. None of it is live yet. OpenAI says it will add support "in the coming months," Google says early 2027, and Microsoft starts on Feb. 1, 2027. Microsoft is also adding a separate page-level setting for "grounding," meaning whether your pages can be used to answer questions in tools like Copilot. The IETF document is still a draft, not a finished standard.

What to weigh: if you run a site, this will eventually turn a long list of per-company blocks into one line, but only with companies that agree to honor it, and it's voluntary. Opting out of training doesn't take you out of AI answers that pull from live pages. That's a separate choice, and Microsoft is adding its own setting for it.