One Line in Robots.txt: What the New AI Training Opt-Out Signal Means for Your Website

Something unusual happened in the AI world this week: the biggest players actually agreed on something. Google and Microsoft have joined OpenAI and Anthropic in backing a single, shared robots.txt signal that lets any website owner say, in one line, that their content should not be used to train AI models.

Until now this was a messy job. Every AI company ran its own crawlers under its own names, and each one needed its own rule. You blocked GPTBot for OpenAI, Google-Extended for Google, ClaudeBot for Anthropic, and then hoped you had not missed one. Some companies operated several bots at once, and the list kept changing. For a small business without a technical team, keeping up was close to impossible.

The new signal promises to replace that whole mess with one line in a file most websites already have. It is a welcome development, but the details matter a lot, and the rollout is slower and patchier than the headlines suggest. Let's sink in.

What exactly was announced

The mechanism comes from an Internet-Draft titled Machine-Readable Training Preference Signal, written by Kevin Kelley of Anthropic and dated 30 October 2026. The draft defines a policy token, __usage_ai-training, that works inside an ordinary robots.txt file. The two underscores at the start are intentional, by the way. They mark the entry as a usage policy rather than the name of yet another crawler.

In early November the companies lined up behind it one by one. Anthropic has already begun honouring the token. OpenAI said its support will arrive in the coming months. Microsoft committed to a firm start date of 1 February 2027, and Google is targeting some point in 2027. Meta says it supports the idea of a common signal but has not committed to honouring it, which is a notable gap given how much crawling Meta does.

It is worth being precise about the status of the document itself. This is an individual draft, not an adopted IETF standard. It has no formal standing yet, and it will lapse in May 2027 unless it is revised. What gives it weight is not the standards process but the public commitments of the four companies behind it.

How the signal works in practice

The syntax reuses what robots.txt already does. You add a group with User-Agent: __usage_ai-training, then put Disallow and Allow lines underneath it, exactly as you would for a normal crawler. A single Disallow: / tells every participating AI company not to use any of your content for model training. You can also keep parts of the site open: Disallow: / followed by Allow: /press-kit/ blocks training on everything except your press materials. Your existing robots.txt rules stay as they are, and the per-company controls like GPTBot, Google-Extended and ClaudeBot keep working alongside the new token.

Two things make this different from blocking a crawler. First, the token governs how your content may be used, not whether it can be fetched. A crawler can still visit your pages, for search indexing for example; it just cannot feed them into model training. A crawler that collects material only for training may simply skip pages it is not allowed to use. Second, the default is permissive. If your robots.txt says nothing, you are allowing training, the same as today. The signal only does something once you add it.

There is one more limitation that catches people by surprise: the token is not retroactive. Content that AI companies collected over the past years stays collected. The opt-out applies to future crawling, not to what already sits inside the models.

Training is not the same as AI search

The draft leans on a narrow definition of training, taken from the IETF's AI Preferences work: training means changing a generative model's learned parameters. Grounding, when an AI assistant fetches your page in real time to answer somebody's question, is a separate activity with separate controls. Models used only for classification or ranking are outside the scope entirely.

This distinction matters commercially. Microsoft keeps a separate page-level AI Grounding preference and warns openly that blocking grounding will significantly reduce your content's visibility in AI-generated experiences. Blocking training alone does not carry that warning. Google went another way and bundled things together: a site using the new token will be treated as having disallowed Google-Extended, which also covers grounding in Gemini apps. So with Google, at least for now, opting out of training also costs you some AI visibility.

For most businesses this is the real strategic question, not the syntax. We wrote the day before about AI Overviews now appearing on more than 80 percent of Google searches. AI answers are becoming a primary discovery channel, and a site that is invisible to them is invisible to a growing share of potential customers. An opt-out that quietly removes you from those answers is not a free choice.

The rollout will be slow and uneven

On paper, four companies support the signal. In practice, only Anthropic honours it today. OpenAI gives itself a few months, Microsoft starts in February 2027, Google at some point during 2027. For most of next year the token will be a statement of intent rather than a working control, so keep your per-bot rules in place and do not delete anything yet.

Meta is the open question. After criticism of commercial terms that allowed it to crawl sites linked to business accounts, the company revised those terms, with the new version due on 20 November, and removed training from the listed uses. Whether Meta will honour the shared signal remains unanswered; the News/Media Alliance's Sam Quigley put it carefully, saying it is still potentially on the table.

And then there is everyone else. Robots.txt is a request, not a wall. The big four have reputations and regulators to think about, but hundreds of smaller crawlers can simply ignore the file, and some do. Cloudflare has blocked AI crawlers by default for new customers since July 2025 and even lets participating sites charge crawlers for access. If enforcement really matters to your business, you still need bot management at the network level, not just a line in a text file.

Why the industry suddenly agrees

The short answer is lawsuits. The New York Times sued OpenAI and Microsoft back in December 2023, and the case still shapes the debate around training data. Anthropic agreed to pay 1.5 billion dollars to settle authors' claims over pirated books used in training. Penske Media sued Google over AI Overviews in 2025. Every AI company now has a strong interest in showing courts and publishers that it respects a clear, machine-readable opt-out.

Regulation pushes in the same direction. The EU's code of practice for general-purpose AI treats robots.txt as the primary machine-readable way for publishers to reserve their rights, and Google has said the new token aligns with it. A shared signal is cheap to support, looks good in a courtroom, and conveniently keeps the default set to allowing training. Site owners get a real tool out of it, but nobody should mistake it for altruism.

What your business should do now

Start with an honest question: what is your content actually worth to a model, and what is AI visibility worth to you? If content is your product, say original research, paid analysis or a news operation, a training opt-out probably makes sense, and you should add the token now even though most companies will only honour it later. If you run a service business, a consultancy or an online store, think twice before blocking anything. Being present in AI answers is increasingly how new customers find you, and with Google the training opt-out currently drags grounding down with it.

The practical checklist is short. Review your robots.txt and note which AI bots you already block. Decide your policy per section of the site rather than for the whole domain; marketing pages and documentation often benefit from AI exposure while premium content does not. If you opt out, add the __usage_ai-training group alongside your existing rules, not instead of them. Treat Microsoft's grounding preference as a separate decision. Then put a reminder in your calendar for early 2027 to verify that Microsoft and Google actually honour the token when their dates arrive.

If your business builds digital products, there is a second angle. Any product that crawls the web, fine-tunes models on scraped content or resells datasets should start respecting the token now, before it hardens into an expectation with legal teeth. Getting this right early is far cheaper than retrofitting compliance after a dispute. And if your site runs on a platform like Wix, check where robots.txt is edited in the dashboard; adding the group takes a few minutes, and it is a sensible thing to bundle into your next SEO review.

Final notes

One line instead of a dozen bot rules is real progress, and the fact that OpenAI, Google, Microsoft and Anthropic back the same mechanism makes it likely to stick around in some form. Just keep its limits in view. It is a preference, not protection; it covers training, not AI search; it starts working at different times for different companies; and it does nothing about content that was collected years ago.

The sensible move for most businesses is to make a deliberate choice rather than inherit the default. Decide what AI systems may learn from, write it down in robots.txt, and revisit the decision once the big players' support actually switches on in 2027. If you want help auditing your robots.txt or thinking through the AI visibility trade-off for your site, that is exactly the kind of work we do every week.