OpenAI Proposes robots.txt AI Training Opt-Out Standard, Google and Microsoft Join

What happened

Google and Microsoft are among the companies signing up to a proposed new website crawling standard from OpenAI, according to TechRadar. The standard would let website owners opt out of AI training on their content through a single line in a site's robots.txt file.

The reporting says the line would apply to Google, Microsoft, OpenAI and Anthropic. It would tell these companies not to use a site's content for training. It would not stop their bots from crawling the site, so the control covers how content is used rather than whether it is accessed.

The coverage does not set out a launch date or a timeline for adoption, so it is not yet clear when the mechanism would be in use across the web.

Why it matters

The story sits at the point where AI development meets the basic mechanics of the open web. For years, robots.txt has been a simple way to tell search engines which pages to index or skip. Extending that logic to AI training gives site owners a clearer, more standard way to express consent, and it could reduce the need for bespoke opt-out requests to each AI provider.

The distinction between crawling and training is significant for leaders in digital transformation. Organisations that publish content, from publishers to government information portals, will need to understand which controls govern access, which govern reuse, and which, if any, are enforced. A signal that is respected by several large providers is only as useful as the trust placed in those providers following it.

The Renascence take

Most readers will focus on the headline names. The more important question is whether a single robots.txt line can carry the weight of a consent decision. Consent has to be legible at the point of decision, and a file that most site visitors never see is a weak interface for it. Treating the standard as a finished answer would be a mistake; it is better understood as a first attempt at a shared control vocabulary.

Opt-out signals only matter when they are clear, consistent and verifiable. Operators should separate the question of who may crawl their content from who may train on it, and document both decisions in plain language. Then they should check how each provider's bots actually behave, rather than assuming a standard is honoured because it has been adopted. Customers and partners judge a brand by whether its stated choices hold up in practice.