Matthew Prince isn’t just fighting Google—he’s accidentally building the case for a balkanized internet. And nobody’s talking about the consequences.
The Headline Everyone Got Wrong
When Cloudflare CEO Matthew Prince issued his September 15 ultimatum last week—block mixed-purpose crawlers by default on ad-supported pages—the tech press framed it as a publisher victory. Finally, someone with teeth was standing up to Google. The villain was clear. The hero was obvious. The narrative was satisfying.
But here’s what nobody’s asking: What happens when the internet’s traffic cop starts deciding who gets to see what?
Because that’s exactly what Cloudflare is about to do. And the implications reach far beyond a Google vs. publishers spat.
The 57% Problem: When Machines Became the Majority
Let’s start with the number that should terrify everyone, not just publishers:
AI agents now generate 57% of web traffic. Humans: 42%.
This isn’t a projection. This is 2026. The machines have already won the traffic war.
But here’s the uncomfortable question nobody’s addressing: If we block the machines, what exactly are we “saving” the web for?
Think about it. When you ask ChatGPT, Perplexity, or Gemini a question and it gives you a synthesized answer, you’re not “not using the web.” You’re using it differently. The information still comes from the web. The sources still exist. The pipeline is just… invisible.
Prince’s framing—that humans are becoming “a rounding error”—assumes human browsing is the only legitimate form of web usage. But is it?
Consider:
- A developer using an AI coding assistant that pulls documentation from 50 sources simultaneously
- A researcher using Perplexity to synthesize medical literature across 200 papers
- A student using Gemini to understand climate data from NOAA, IPCC, and academic journals
These aren’t “bots destroying the web.” These are humans using tools to access information more efficiently. The traffic is machine-generated, but the intent is human.
By blocking crawlers indiscriminately, Cloudflare isn’t just stopping Google. It’s potentially breaking the infrastructure that makes modern knowledge work possible.
The Publisher Paradox: Why Opt-Outs Might Kill the Very Thing They Protect
The UK Competition and Markets Authority’s order—letting publishers opt out of AI features while staying in search—sounds reasonable. Fair, even. But it contains a logical trap that could backfire spectacularly.
Here’s the paradox:
AI Overviews reduce clicks. But they also increase visibility.
Google’s own data shows that brands cited in AI Overviews earn approximately 120% more clicks per impression than uncited brands. Reach Media lost 50% of referral traffic? True. But how many of those users would have clicked anyway in a world without AI summaries? And how many new users discovered Reach’s content because it was cited in an Overview?
We don’t know. Because nobody’s measuring assisted discovery—the user who sees your brand in an AI summary today, searches your site directly tomorrow, and subscribes next week.
Small publishers have lost 60% of search traffic over two years? Also true. But correlation isn’t causation. Algorithm updates, competition from Reddit and LinkedIn, short-form video, newsletter fatigue, economic pressures on ad budgets—all of these factor in. Blaming AI Overviews alone is convenient. It might not be accurate.
The deeper danger: If publishers opt out en masse, they don’t just lose AI traffic. They risk becoming invisible to the next generation of search entirely.
Because here’s what the opt-out advocates aren’t saying: AI-native search—ChatGPT Search, Perplexity, Gemini, Bing Copilot—isn’t a side feature anymore. For millions of users, especially younger ones, it’s becoming the search. If your content isn’t in these systems, you don’t exist to a growing segment of the internet.
Opting out of AI isn’t like opting out of a newsletter. It’s like opting out of being findable.
The Crawler Conspiracy Nobody’s Mentioning
Let’s talk about the real elephant in the room.
Googlebot sees 3.2x more of the web than OpenAI’s crawler and 4.8x more than Microsoft’s. Prince’s numbers. Undisputed.
But here’s the question: Why?
It’s not because Google is sneakier. It’s because Google has been the web’s librarian for 25 years. Publishers have spent decades optimizing for Googlebot. They’ve built sitemaps, structured data, robots.txt files, and entire SEO industries around Google’s crawler. Of course Google sees more. It was invited in.
OpenAI and Microsoft? They’re newcomers. Publishers haven’t optimized for them. Many don’t even know their crawlers exist. GPTBot and Bingbot aren’t “less invasive”—they’re just less established.
Prince’s ultimatum doesn’t level the playing field. It cements Google’s advantage. Because if Cloudflare blocks “mixed-purpose crawlers” by default, who do you think has the engineering resources, legal teams, and publisher relationships to negotiate exceptions?
Hint: Not the startups. Not the open-source projects. Not the non-profits.
Google.
The ultimatum that claims to hurt Google might actually entrench it further by creating a two-tier web: one where Google can negotiate access, and everyone else gets blocked at the door.
The Internal Slides: Why Google’s “Monetization” Admission Matters More Than You Think
The most damning revelation in this whole story isn’t the traffic numbers. It’s a line from internal Google slides, disclosed in a US antitrust case:
Google considered offering an AI opt-out before launching AI Overviews but chose not to because it was “evolving into a space for monetisation.”
Read that again.
Google knew publishers would want to opt out. Google knew the bundling of search and AI crawling was controversial. And Google deliberately chose revenue over transparency.
This isn’t a technical decision. It’s a business strategy to extract value from content creators without sharing it. The same content that makes Google’s AI useful is the same content Google is now using to reduce traffic to the creators who made it.
That’s not “innovation.” That’s enclosure of the digital commons.
And it’s why the publisher revolt isn’t just about traffic. It’s about the fundamental contract of the web: I let you index my content, you send me visitors. Google broke that contract. Now everyone is scrambling to rewrite the terms.
Three Scenarios for What Happens Next
Scenario 1: The Balkanized Web (Most Likely)
Cloudflare blocks mixed crawlers. Google negotiates exceptions. OpenAI, Perplexity, and smaller AI companies get shut out. The web fragments into Google-accessible and Google-inaccessible zones. Innovation in AI search stalls. Users get worse answers. Publishers get less traffic overall, not more.
Scenario 2: The Licensed Web
Publishers start charging AI companies for access. Google, Microsoft, and OpenAI pay licensing fees. Large publishers (NYT, Reuters, AP) get deals. Small publishers get nothing. The information gap between rich and poor media widens. The open web dies a little more.
Scenario 3: The Transparent Web (Least Likely, Best Outcome)
Google separates its crawlers. Publishers choose search indexing, AI training, both, or neither with granular control. AI companies disclose what they crawl, how they use it, and how they attribute. A new standard emerges—maybe even a protocol—for ethical AI content usage. The web stays open, but with clearer rules.
Which one happens depends on whether regulators, infrastructure providers, and AI companies can agree on something they’ve never agreed on before: shared rules for a shared resource.
What This Means for You: Three Takeaways
1. If you publish content, don’t opt out blindly.
Yes, protect your work. But understand that AI visibility is becoming the visibility. Consider negotiating attribution requirements instead of blanket blocks. Demand citation. Demand links. But don’t disappear from the systems where the next generation discovers information.
2. If you build AI products, crawl transparently.
Publish your crawler policies. Respect robots.txt. Cite sources prominently. Build attribution into your product, not as an afterthought. The companies that do this will win publisher trust—and better content—over the long term.
3. If you’re a user, pay attention to who you trust.
When you ask an AI a question, where does the answer come from? If the source is invisible, the accountability is too. Demand transparency. Use AI tools that cite sources. The future of reliable information depends on users caring about the pipeline, not just the output.
The Bottom Line
Matthew Prince’s ultimatum feels like a stand for the open web. It might be. But it might also be the first step toward a web where access is negotiated by gatekeepers, not governed by open protocols.
Google’s crawler bundling was a mistake—maybe even a cynical one. But the solution isn’t another gatekeeper. It’s transparency, choice, and shared standards.
The internet wasn’t built on ultimatums. It was built on protocols. If we forget that, we don’t save the web. We replace one monopoly with another.
And the next monopoly might not even pretend to be on your side.
What’s your take? Is Cloudflare’s ultimatum a necessary defense or a dangerous precedent? Let me know in the comments.
Recommended for you:
Google Starts Labeling AI-Generated Ads: Why This Matters More Than You Think
