Publishers, including Conde Nast, TIME and The Associated Press, have joined Cloudflare in blocking AI bots from crawling their sites for free|Cloudflare|Facebook
Cloudflare, which operates around 20% of the web, launched Pay per Crawl on Tuesday—a marketplace that lets websites charge AI companies like OpenAI and Anthropic for scraping their content.
The tool, still in the private beta phase, allows websites to set payments for every single AI bot visit.
For a long time, AI bots like ChatGPT or Google’s AI tools have been using content for free to learn and answer users’ questions, reducing site clicks and visits. This means website owners, especially news publishers, don’t get visitors, thus affecting their revenue.
Cloudflare found in June that for every referral it gave them,
-
Google scraped websites 14 times.
-
OpenAI scraped them 1,700 times.
-
Anthropic scraped a whopping 73,000 times.
It matters because more people are using AI to find answers instead of searching websites.
Cloudflare hopes its tool helps even the playing field, specifically for smaller websites that haven’t landed big-money AI deals.
Publishers, including Conde Nast, TIME, The Associated Press, The Atlantic, Adweek, and Fortune, have joined Cloudflare in blocking AI bots from crawling their sites for free, unless permission is explicitly granted.
The company hopes its system will give publishers more control and generate revenue in the AI era, as traffic from Google declines and chatbots answer most web user queries.