Meta Wants to Crawl Your Website: What the New Commercial Terms Mean for Your Business

If your business runs a Facebook page, Instagram ads or a WhatsApp Business channel, there is a date you should put in the calendar. On 30 October 2026 Meta's updated Commercial Terms come into force, and tucked into Section 1 is a clause that gives Meta the right to crawl websites connected to your business and use what it finds to train its AI.

The change was first noticed by Nieman Lab in mid-October, and the reaction was quick. Meta insists the new wording is just a tidy-up of language that already existed across different documents. Lawyers who actually read the text see something considerably broader, and some business owners are already talking about closing their Meta accounts before the deadline.

There is no button to press to decline. If you keep using Meta's business products after 30 October, you have agreed. So it is worth understanding what exactly you are agreeing to. Let's sink in!

What the new Commercial Terms actually say

The update replaces the previous version of the Commercial Terms, which dated back to March 2024. The key addition sits in Section 1. It says Meta may use automated software, including web scrapers, bots and crawlers, to access, index, cache, collect or crawl domains and pages that are associated with your products or services, or that are made available through your use of Meta products. It may also create test credentials to get into those pages, and come back periodically to pull updates.

Then comes the part that matters most. Content collected this way can be used to provide, develop, improve, research and train Meta's products, and the terms define AI at Meta as one of those products. Put the two together and the plain reading is simple: Meta can crawl the website linked to your business account and train its AI models on your content.

The crawl clause is not the only change. The new terms add a dedicated AI section, supply Meta's services on an as-is basis, widen the indemnity you give Meta, cap Meta's liability at the greater of 100 dollars or the fees you paid over the previous twelve months, and push US businesses into individual arbitration.

Who is affected and why "associated with" matters

The Commercial Terms cover anyone who uses Meta's products for business: managing a Facebook Page or group, running ads, sending marketing messages, operating a shop, or simply using the Meta Business Suite dashboard. That is a very wide net. Almost every company with any social presence falls into it, from a local service business to a large publisher.

The phrase doing the heavy lifting is "associated with", and the terms never define it. Your Page links to your website, your ads have landing pages, your product catalogue points at product pages, your WhatsApp Business profile carries a link. All of that is plausibly associated with your products or services. Where the association stops, nobody knows.

The test credentials wording worries legal experts most. James Grimmelmann of Cornell Law said it could let Meta sign up as a fake user to get past registration walls, and Meta declined to answer whether those credentials could be used to bypass paywalls. If part of your site sits behind a free sign-up form, you should assume a crawler with test credentials can reach it.

Meta's defence and the lawyers' reading

Meta's position is that critics are overreading the clause. Spokesperson Andy Stone said on 16 October that the language aims to pull accurate product information from third-party sites for ads and content recommendations, not to scrape news sites, and that the update merely consolidates wording that was previously spread across several documents, including the generative AI terms for ads.

That older wording, though, allowed crawling of ad URLs only for those ads, and said nothing about training. The new clause is broader on both counts. Matt Topic, a media and IP attorney at Loevy & Loevy, called it a remarkable provision to bury in terms of service, and added that while it is not clear a court would enforce it, businesses should assume it would be enforceable.

Benjamin Sobel of the University of Wisconsin compared the setup to tying in antitrust law, where access to one product is conditioned on accepting another. The parallel with Google is hard to miss: publishers had to accept AI Overviews to appear in search until an opt-out finally arrived in August 2026. With Meta there is no partial opt-out at all. The official advice for those who disagree is to stop all business use of Meta products.

The robots.txt question

The new terms do not mention robots.txt at all. Meta says content owners can still manage how their content is accessed with it, and its developer documentation explains how to block each crawler. The training crawler is meta-externalagent, and according to Meta it respects robots.txt rules. The Known Agents tracker says about 19 percent of top websites already block it.

Meta's crawling footprint is growing fast either way. Cloudflare data shows Meta's share of AI crawler traffic rose from 0.9 percent in July 2024 to 7.5 percent in July 2025. Meta also runs meta-webindexer, which feeds Meta AI search and citations, meta-externalads for its advertising products, and fetchers like facebookexternalhit that build link previews and may bypass robots.txt for single requests.

The uncomfortable detail is legal rather than technical. Robots.txt is a request, not a contract. Several lawyers pointed out that Meta could argue the Commercial Terms you accepted override your blocking instructions. Nobody has tested that in court, so for now a robots.txt rule is your best available signal, not a guarantee.

What this means for your marketing and SEO

For e-commerce businesses the picture is mixed rather than purely bad. Two days before the terms take effect, on 28 October, Meta starts in-app product browsing inside existing ads. A crawler that keeps your product data fresh can make those ads more accurate, which is presumably the use case Stone was describing. If your site is a catalogue and your revenue runs through Meta ads, blocking everything may hurt more than it protects.

Content businesses sit at the other end. If the value of your site is the writing, data or research on it, the training clause means that value can flow into Meta's models with nothing coming back. And there is the same trade-off that search introduced: block meta-externalagent and you reduce training exposure, but block meta-webindexer too and you lose the chance of being cited in Meta AI answers.

European businesses have an extra layer. EU copyright law lets rightholders reserve their content from text and data mining in machine-readable form, and since August 2025 the AI Act requires general-purpose AI providers to respect those reservations. How a US contract interacts with that framework is an open question, and Meta has not said Europe is excluded from the new terms.

Practical steps to take before 30 October

First, decide your stance deliberately rather than by default. If you do not want your content training Meta's models, add a rule for meta-externalagent to your robots.txt with a disallow for the whole site. Leave facebookexternalhit alone so link previews on Facebook and WhatsApp keep working, and think twice before blocking meta-webindexer if Meta AI citations matter to your traffic.

Second, audit what is actually associated with your business account: the website field on your Page, every ad landing page, your product catalogue URLs and the link in your WhatsApp profile. That list is the realistic crawl surface. If you use Cloudflare, check its AI bot settings too, since it may already be blocking Meta's training crawler on your behalf.

Third, protect anything genuinely valuable with real access controls. Given the test credentials wording, a free registration form is not protection. Paid content, client portals and anything commercially sensitive should sit behind proper authentication. And on 30 October, save a copy of the terms you are accepting, so you know exactly which version you agreed to if the wording shifts again.

If you are building a new site or digital product, treat crawler policy as part of the launch checklist, the same way you treat analytics or cookie consent. Deciding at design time which directories are public, which are gated and what goes into robots.txt is cheap. Retrofitting that decision after your content is already inside a training set is not.

Final notes

The honest summary is that the terms read broad, Meta says they are narrow, and nobody will know who is right until someone litigates. Waiting for that answer is not a strategy. A robots.txt rule, an audit of your linked pages and proper access controls cost a few hours and keep most of your options open.

Meta is betting that businesses need its distribution more than they fear its crawlers, and for most businesses that bet is probably right. That makes it all the more sensible to take the free protections while they are available. The deadline is 30 October, and doing nothing is also a choice.