After earlier this year launching a marketplace that allows websites to charge AI bots for scraping their content, web infrastructure provider Cloudflare is pushing for increased regulation in the AI sector.
The company’s chief executive, Matthew Prince, says he’s in London to speak with the U.K.’s Competition and Markets Authority (CMA), where he’s proposing stricter rules on how Google should be able to compete in the AI race, given its search dominance.
The CMA earlier this month designated Google with a special status in the search and advertising markets because of its “substantial and entrenched” position. The move will allow the regulator to impose more stringent regulations beyond just search and ads, including in areas like Google’s AI Overviews and AI Mode, the Discover feed, Top Stories, and News tab.
According to Prince, Cloudflare is in a good position to make recommendations because it’s not in the AI business itself but has a large number of relationships with the AI companies.
“We don’t have a dog directly in the fight. We’re not an AI company,” Prince said, speaking at the Bloomberg Tech conference in London this week. “We’re not a media publisher, but we’re this network that sits between them — 80% of the AI companies are our customers,” he added.
The Cloudflare boss believes that Google should have to compete on the same footing as other AI companies, which is not what it’s doing today, he said. Rather, Google uses its existing web crawler to crawl content for its AI products and services, in addition to its search engine. This, Prince said, gives Google an unfair advantage. (Prince is referring to Googlebot, which crawls for Search, including its AI features, like AI Overviews. A Google spokesperson, Ned Adriance, said that site owners can opt out of having content used for training AI products with Google Extended, which doesn’t impact a site’s inclusion in Google Search. Some media companies would likely prefer to fully opt out of AI features, however, so the point largely stands.)
“Google is saying, ‘We have an absolute God-given right to all of the content in the world, even if we don’t pay for it, because look what we did for the last 27 years,’” Prince explained. “And they’re saying we can take it and use the same crawler we use for search in order to power our AI systems. And if you want to opt out of one, you have to opt out of both,” he noted.
This, obviously, is not feasible for most, particularly those in the media business, where losing search means losing about 20% of your revenue, the executive said.
“But it gets even worse. If you block Google’s crawler, it blocks their ad safety team, which means that your advertisements across all of your platforms stop working, which is just a non-starter,” Prince added.
Because Google bundles its crawler, it’s able to get access to content that others, like Anthropic, OpenAI, and Perplexity, would have to pay for.
“The problem is that we then will have effectively handed the game to Google,” he said.
The solution, Prince said, is to foster a lot of competition in the market, where potentially thousands of AI companies could compete to buy content from thousands of media businesses and millions of small businesses. He suggested that what the U.K.’s CMA was doing by flagging Google as a potential regulatory target was a thoughtful move, and one that indicates they’re aware of Google’s unique advantage.
Cloudflare has also provided the CMA with data that shows how Google’s crawler works and why it’s nearly impossible for other players to replicate the same success Google could have.
Prince isn’t the only one to share these opinions in recent days. Last month, Neil Vogel, CEO of People, Inc., the largest digital and print publisher in the United States, which operates over 40 media brands, said essentially the same thing. In an interview, he called out Google as a “bad actor,” saying that media companies had no choice but to let Google crawl their sites for AI content because of the way the crawlers were combined.
Vogel, whose company had adopted Cloudflare’s solution to block AI crawlers that don’t pay, claimed the system was working, as he said deal discussions were underway with several large LLM providers.
Updated with Google comment.
