Should Publishers Block AI Crawlers? What the robots.txt Data Says About the Trade

The robots.txt file was designed for a simpler question: may this crawler index my site so a search engine can send people to it? The answer was usually yes, because indexing led to traffic. AI crawlers broke the bargain. Some of them fetch a page to train a model that will never send anyone back. Some fetch it to compose an answer that may cite you. Some fetch it on behalf of a user who asked for it. They arrive with different names, and publishers have spent two years deciding, one user-agent at a time, which to let in.

This report assembles the measurements: who blocks what, how well blocking works, what the different crawlers give back, and what the licensing alternative has actually produced. Then we say what we do and why.

1. Who blocks what

BuzzStream’s April 2026 audit examined the robots.txt files of the 100 largest news sites in the U.S. and U.K. by traffic, checking for disallow rules against eleven AI-related crawlers. The headline: 79% block at least one training bot; 14% block all eleven; 18% block none. The more interesting figure is that 71% also block at least one retrieval bot — a crawler that fetches pages to answer live queries and, in doing so, is the mechanism by which a publisher gets cited.

Source: BuzzStream, Which News Sites Block AI Crawlers in 2025? (Nero, April 8, 2026); 50 U.S. and 50 U.K. news sites by SimilarWeb traffic.

Two details stand out. U.S. publishers block Google-Extended — the token that opts a site out of Gemini training without affecting Google Search — at twice the U.K. rate (58% versus 29%). And OAI-SearchBot, the crawler OpenAI uses specifically to surface and link sources in ChatGPT search, is blocked by about half of the sites, often by the same rules written to stop GPTBot two years earlier.

The pattern is wider than news. Cloudflare’s 2025 year in review found GPTBot, ClaudeBot, and CCBot the most frequently fully disallowed crawlers across its network, while Googlebot and Bingbot — which crawl for both search and AI — drew partial disallows rather than site-wide blocks. By December 2025, the Register reported roughly 5.6 million sites disallowing GPTBot and 5.8 million disallowing ClaudeBot, both up around 70% since July.

2. Blocking is porous

A robots.txt rule is a request. The measurements of how often it is honored are not reassuring.

TollBit, which sits between publishers and bots for a network of sites, reported that the share of AI scrapes bypassing robots.txt rose from 3.3% in the fourth quarter of 2024 to about 13% by the end of the first quarter of 2025 — more than 26 million non-compliant fetches in March 2025 alone — and that retrieval-driven scraping had overtaken training scraping on its network. The Tow Center’s March 2025 citation study added a pointed example: Perplexity correctly identified nearly a third of ninety excerpts drawn from publishers that had explicitly blocked it.

Source: TollBit, State of the Bots, Q1 2025, as reported by FIPP and The Media Copilot.

Enforcement at the edge is the response. Cloudflare made AI-crawler blocking the default for new domains on July 1, 2025, and reported that customers had blocked more than 416 billion AI bot requests by early December. Its 2026 revision of the defaults is instructive: on pages that carry advertising, crawlers in the “training” and “agent” categories are blocked by default, and crawlers in the “search” category are allowed. The company that built the largest AI-blocking apparatus on the web drew the line between training and retrieval, not between AI and everything else.

3. What each kind of crawler gives back

The reason to draw that line is in Cloudflare’s own traffic data. Across 2025, training crawls ran at seven to eight times the volume of search crawls and, at peak, 32 times the volume of “user action” crawls — fetches made on behalf of a person asking an assistant a question. Search crawling ended the year about 10% below where it started; user-action crawling grew more than twenty-fold.

The crawl-to-refer ratio — pages fetched per visitor sent back — is the sharpest single measure of the bargain, and it varied enormously by company and by month:

Operator	2025 crawl-to-refer ratio (Cloudflare Radar)	Character
Google (Search)	roughly 3:1 to 30:1	Crawls to send traffic
Microsoft (Bing)	roughly 50:1 to 70:1	Search, with AI features
Perplexity	under 100:1 to 700:1; generally under 400:1 by year-end	Retrieval-first
OpenAI	up to 3,700:1 at peak; improving as search features grew	Training plus retrieval
Anthropic	25,000:1 to 500,000:1, erratic early in the year	Training-dominant

The ratios are volatile, and secondary reports citing a single month’s figure should be read with that in mind. But the ordering is stable and it maps directly onto business model: a crawler whose product is an answer with sources has a reason to send you traffic; a crawler whose product is a trained model does not.

What that traffic amounts to today is small. Chartbeat data reported by Axios in March 2026 shows ChatGPT referrals to publishers up more than 200% year over year — and AI chatbots combined still accounting for less than 1% of publisher pageviews, against a 34% decline in Google Search referrals and 15% in Discover over the same period. Blocking a retrieval bot costs you a citation that is currently worth little in visits. Allowing it costs you nothing you were going to keep. The question is which of those numbers you expect to grow.

4. The licensing alternative

The strongest argument for blocking is leverage: content withheld can be sold. That has worked for some. Publishers have signed licensing agreements with OpenAI, Amazon, and others, and Google has begun paying partners under programs tied to citation features.

The Reuters Institute’s survey of 280 media leaders puts the expectation in proportion. Getting platforms to pay for content was the most-cited new growth opportunity for 2026, named by 37% of commercial publishers. Asked what it would actually yield, 20% expected significant income, 49% a minor contribution, and 20% nothing at all.

Source: Reuters Institute, Journalism, Media, and Technology Trends and Predictions 2026; 280 media leaders in 51 countries.

For the handful of publishers with archives large and distinctive enough to license, withholding is a negotiation. For everyone else, it is a policy with a cost and no counterparty.

5. A decision, not a switch

The data supports treating the three kinds of crawler differently, and the tools now allow it.

Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, meta-externalagent): these fetch to build models and, by the ratio data, send almost nothing back. Blocking them is a reasonable default for a publisher who has, or expects to have, something to license — and a legitimate statement of terms for one who does not. It is also the category most likely to be fetched regardless, so pair the rule with edge enforcement if it matters to you.

Retrieval crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot and Perplexity-User, Claude-SearchBot and Claude-User, Google’s search crawler with AI Overviews and AI Mode, Bing): these fetch to answer a live question and can cite you. Blocking them removes you from answers without removing the answers. For a publisher whose value is specific, verifiable information, this is where the growth is and where a citation is measurable.

Agent crawlers (fetches made by an assistant acting for a user — booking, comparing, buying): a newer category, growing fastest, and the one where the terms are least settled. Cloudflare blocks these on ad pages by default; a publisher with a transactional product may want them.

Whatever the policy, the two things that make it a decision rather than a reflex are separating the categories and measuring the result: referrals and citations from each AI surface, tracked alongside search, so that next year’s robots.txt is written from data.

Methodology and caveats

Blocking rates are from BuzzStream’s audit of 100 news sites (50 U.S., 50 U.K., selected by SimilarWeb traffic) published April 8, 2026; robots.txt directives are what a site declares, not what it enforces. Cloudflare’s crawler and ratio figures are from its 2025 Radar year in review (data through December 2, 2025) and cover traffic on Cloudflare’s network; ratios are ranges over the year and are highly volatile month to month. The Register’s site counts for GPTBot and ClaudeBot disallows are as reported in December 2025. TollBit’s compliance figures are from its Q1 2025 State of the Bots report as summarized by FIPP and The Media Copilot; TollBit’s network skews to publishers who have adopted its tooling. Tow Center figures are from its March 2025 citation study. Chartbeat referral figures are from data provided to Axios (March 17, 2026) covering thousands of mostly news and media sites. Reuters Institute figures are from a non-probability survey of senior leaders.

What to do about it

1. Write rules by category, not by vendor. Group user agents into training, retrieval, and agent, and decide each group on its own terms. A rule written for GPTBot in 2023 should not still be blocking OAI-SearchBot by accident.
2. If you block training, enforce it at the edge. Roughly one request in eight ignored robots.txt by early 2025. A CDN or bot-management rule is what makes a training block real.
3. Let retrieval crawlers in unless you have a specific reason not to. They are the path to citations, the citation data favors specific, sourced pages, and the referral numbers — small today, growing fast — are the ones you want to be measuring.
4. Publish a plain-language map for assistants. A /llms.txt file and consistent structured data make it easier for retrieval systems to find and attribute your best pages.
5. Instrument the outcome. Track referrals by AI source and citations by page type. Revisit the policy annually with the numbers in hand.
6. Negotiate from data, not from absence. If licensing is realistic for you, the evidence of what your content contributes to answers is a stronger position than a wall.

Sources

1. BuzzStream — Which News Sites Block AI Crawlers in 2025? (Nero, April 8, 2026) — 100 U.S. and U.K. news sites; robots.txt audit of 11 crawlers
2. Cloudflare — The 2025 Cloudflare Radar Year in Review (December 2025) — Crawl-to-refer ratios, crawler purpose volumes, most-blocked bots
3. Cloudflare — Your site, your rules: new AI traffic options for all customers (2026) — Default blocking of training and agent crawlers on ad-supported pages; search allowed
4. The Register — Publishers say no to AI scrapers, block bots at server level (December 8, 2025) — Site counts disallowing GPTBot and ClaudeBot; Cloudflare's 416 billion blocked requests
5. TollBit — State of the Bots reports — robots.txt bypass rates, Q4 2024 to Q1 2025
6. FIPP — State of the Bots: FIPP and TollBit put the spotlight on AI scraping
7. The Media Copilot — 'The pipes are leaky': New report shows AI scrapers bypassing publisher protections at scale — 26 million non-compliant scrapes in March 2025
8. Columbia Journalism Review / Tow Center — AI Search Has a Citation Problem (March 6, 2025) — Perplexity identifying excerpts from publishers that blocked it
9. Search Engine Journal — Search Referral Traffic Down 60% For Small Publishers, Data Shows (reporting Chartbeat data via Axios, March 2026) — ChatGPT referrals +200%; chatbots under 1% of pageviews; Google Search −34%, Discover −15%
10. Reuters Institute — Journalism, Media, and Technology Trends and Predictions 2026 (January 2026) — Licensing expectations