This session was delivered in Spanish, recorded by an AI recorder, and translated into English by AI. We have made every effort to ensure accuracy.
Chema Alonso’s first conference talk was in Barcelona, on Windows XP, in a suit that didn’t fit because he had packed neither a belt nor socks. On the 7th of September, he was back in the City, at CaixaForum for Media Party Barcelona, as Vice-President and Head of International Development at Cloudflare, the company he joined from Telefónica in August 2025. This time he spoke to journalists about who gets paid when machines, rather than readers, consume what they write.
By Alonso’s account, the network runs in more than 330 cities, fronts around a quarter of internet domains and a similar share of global traffic, and blocks some 250 billion attacks a day. Cloudflare’s own published figure is more than 20% of the web. Either way, few companies see more of the web, and this year that view showed automated traffic overtaking human traffic for the first time.
The traffic that paid for content has stopped arriving.
For two decades the web ran on an exchange: publishers gave crawlers their content, search engines sent back readers, and readers saw ads or bought subscriptions. AI platforms take the content and return agents, which do not click. “AI platforms are taking the data for agents that are not being redirected to websites where they don’t click or see ads,” Alonso said. The pact must be rewritten so that “valuable content continues to be valued and paid for.”
The obstacle is opacity: crawlers can index for search, feed inference, or model training, and many do not say which. Alonso put the share that declares no purpose at 38%, with 17% identifying as search. Cloudflare’s July report cut the data differently: 52% of crawler requests on its network are now for AI training, up from 22% in spring 2025, and more than a third come from mixed-use bots that blend search, agents and training. In a Europe that talks about data sovereignty, he noted, a publisher might reasonably want to know whether its pages are training a model it never agreed to feed.
Publishers have a tool for this and are afraid to use it. Among the 10,000 highest-traffic domains, the share whose robots.txt disallows training crawlers is tiny. “People are afraid of blocking,” Alonso said, because the crawler they most need to block is also the one that keeps them in search.
Spain is 22nd, and Alonso treats that as a diagnosis.
He ran the numbers live on Cloudflare Radar, the company’s public traffic dashboard. Ranked by AI-related traffic, the United States leads, followed by Germany and the Netherlands. Spain sits at 22nd. “We are not even on page two,” he told the room, reading it as a measure of how little AI has penetrated Spanish companies and daily life. His own caveat applied: bots make thousands of requests per interaction, so traffic volume says more about machine activity than about the number of bots.
Radar also hosts Cloudflare’s directory of verified bots and their stated purposes. “If necessary, we call the bots by phone,” Alonso said.
Four announcements, and a default that flips on Tuesday
Cloudflare declared its first Content Independence Day on 1 July 2025, blocking AI training crawlers by default on new domains and launching Pay Per Crawl. The second, on 1 July this year, brought four changes.
The first is a demand for transparency. Crawlers are now classified as Search, Agent or Training, and site owners can allow, block or charge each class separately. Alonso listed what Cloudflare is asking of AI companies in return: honour a training block; offer a way to have content removed from a model; give URL-level visibility of whether a page was used for training, search or inference; publish metrics comparable to Search Console data; and guarantee that blocking training carries no search penalty.
The second is data for publishers: how much bot traffic reaches a site, which URLs are being collected, and a crawl-to-referral ratio showing how often a platform takes content against how often it sends a reader.
The third is a new payment model. Pay-per-crawl paid a publisher when a bot fetched a page. Cloudflare has now declared that insufficient and is moving to pay per use, in which the publisher is paid when content appears in an answer. Two partners are live. Ceramic.ai, the AI search company founded by Anna Patterson, pays for non-paywalled news each time it surfaces in results. You.com pays for paywalled premium content each time it is used in a response.
The fourth is efficiency. Cloudflare’s data shows more than half of AI crawl traffic re-fetches pages that have not changed, at the publisher’s expense. Alonso described a freshness signal that tells a crawler whether a page has changed, delivery of pages as Markdown rather than rendered HTML, and link maps built from real navigation. A pilot with OpenAI, announced in July, is defining APIs to carry site ratings, URL maps, freshness and rate of change; Cloudflare says it covers search only and shares no content for training.
Alonso did not dwell on the deadline. From Tuesday 15 September, Cloudflare’s defaults change: on pages that carry advertising, Training and Agent crawlers are blocked unless the site owner decides otherwise, and a crawler that mixes purposes is judged by its most restrictive behaviour. That covers new domains and free-tier customers, and it is why the industry has spent the summer arguing about Googlebot, which does not separate discovery from AI use.
A network operator is writing the rules for a market it will sit inside.
Cloudflare’s position is that transparency creates scarcity, and scarcity gives publishers the leverage to strike licensing deals. Its own report counts more than 50 publisher-AI agreements since 2023 and concedes that licensing remains bespoke and will not replace the referral and advertising revenue it is meant to offset.
The company is also a vendor. Every tool Alonso described runs through Cloudflare’s network, and a market in machine-read content with Cloudflare as clearing house is a business, however sound the argument for it. Pay Per Crawl was the model of the future for exactly twelve months. The list of asks to AI platforms is a list of asks, and Google has spent a year declining the central one.
Alonso closed by saying Cloudflare would keep going “so that the inequality that is happening does not occur, where businesses based on content generation are disappearing or falling to historic lows.” On Tuesday, the defaults flip. What the largest crawler on the web does about it is the part of the pact Cloud
This is massive: “52% of crawler requests on its network are now for AI training, up from 22% in spring 2025, and more than a third come from mixed-use bots that blend search, agents and training.”