Cloudflare, the cloud infrastructure giant powering 20% of the web, has launched a new experimental marketplace called Pay per Crawl, aimed at reshaping how AI companies interact with website content — and giving publishers more control in the process.
Over the past year, Cloudflare has rolled out tools for publishers to combat the surge of AI crawlers, including a one-click AI bot blocker and dashboards that reveal crawler activity. Cloudflare CEO Matthew Prince previously hinted that these were building blocks for a larger platform where content creators could monetize AI access to their data.
Now, that vision is taking shape.
Launched in private beta on Tuesday, Pay per Crawl allows website owners to decide how AI crawlers engage with their sites — setting individual access rates, allowing free access, or blocking them entirely. Each “crawl” can carry a micropayment, with Cloudflare facilitating the transaction. Publishers will also be able to identify the purpose of the crawl, whether for AI training, chatbot answers, or general indexing.
The model could introduce a new revenue stream for publishers struggling with declining traffic from traditional search engines and the growing dominance of AI chatbots. Major outlets like The New York Times have taken legal action against tech companies scraping content without permission, while others have opted for multi-year licensing deals. Yet, these agreements remain exclusive to large players and their profitability remains unclear.
Cloudflare hopes to change that by offering smaller publishers a chance to define their own pricing structure. As part of this effort, all new websites set up through Cloudflare will now block AI crawlers by default, giving domain owners the choice to opt in. The company has received backing from publishers such as TIME, The Atlantic, The Associated Press, Conde Nast, Fortune, and ADWEEK, who support the permission-based model of web crawling.
This shift comes as data shows AI companies are scraping content far more often than they are driving traffic. According to Cloudflare, in June, Google’s crawler scraped its sites 14 times per referral. OpenAI’s crawler did so 17,000 times, and Anthropic’s a staggering 73,000 times per referral.
Both OpenAI and Google are working on “AI agents” that can browse websites on users’ behalf, returning results without the user ever visiting the source — potentially cutting publishers out of the traffic loop entirely. Cloudflare believes Pay per Crawl could evolve into a network-level, automated paywall in such an “agentic” future, where digital agents retrieve licensed content based on user needs and budget.
For now, participation requires both AI companies and publishers to have Cloudflare accounts. Each can define their buy/sell rates for crawling, while Cloudflare acts as the intermediary. According to spokesperson Ripley Park, no cryptocurrency or stablecoins are being used for these transactions, despite speculation that digital payments could streamline such a system.
While the concept is ambitious, its success hinges on publisher buy-in and AI firms’ willingness to pay for content they’ve so far accessed for free. Still, as the dynamics between content creators and AI platforms evolve, Cloudflare appears uniquely positioned to push this conversation — and business model — forward.