Cloudflare’s latest data offers one of the clearest snapshots yet of how AI companies consume the web, and how little they give back.<\/p>\n
\n Loading audio narration…\n <\/p>\n
The company, which powers roughly 20% of the internet, tracks how AI bots crawl websites versus how often those platforms send users back through referrals. The resulting “crawl-to-refer” ratio is a simple yet telling metric: how much value is extracted compared to returned.<\/p>\n
The early April 2026 figures are stark. Anthropic is the worst by a wide margin, with a ratio of 8,800 to 1. That means its bots crawl webpages 8,800 times for every referral sent.<\/p>\n
OpenAI follows at 993 to 1. Microsoft, Google, and DuckDuckGo look far more balanced by comparison.<\/p>\n
Anthropic’s position is particularly striking given its reputation for being “ethical.” That reputation has made it a preferred choice among some users who want to support more responsible AI development. This data highlights a different dimension of ethics \u2014 how companies interact with the broader web ecosystem that provides information for AI model outputs.<\/p>\n
Historically, the internet operated<\/a> on an implicit bargain: websites allowed search engines to crawl and index their content for free, and in return received traffic they could monetize. Generative AI breaks that bargain. Chatbots increasingly provide direct answers, reducing the need for users to click through to original sources.<\/p>\n This results in a system that extracts more value than it gives back \u2014 and in some cases, increases costs<\/a> for site owners due to heavy bot activity.<\/p>\n Anthropic has previously questioned Cloudflare’s<\/a> methodology and pointed to growing referral traffic from new features. Still, the broader trend is hard to ignore. I asked Anthropic for comment this time, too, and it did not respond.<\/p>\n If the web’s economic engine depends on traffic and referrals, these ratios raise a fundamental question: What will incentivize the sharing of verified information online in the future?<\/p>\n Cloudflare is trying with a new marketplace for web content. It’s unclear if efforts like this will succeed. After all, what’s better than using people’s content for free?<\/p>\n Sign up for BI’s Tech Memo newsletter here<\/a>. Reach out to me via email at abarr@businessinsider.com<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"Cloudflare’s latest data offers one of the clearest snapshots yet of how AI companies consume the web, and…\n","protected":false},"author":2,"featured_media":600497,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[62,9359,1930,276,277,49,48,15445,1329,227337,227336,22208,244,194069,47768,61,7479,1890,32786,343,51284],"class_list":{"0":"post-600496","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-artificial-intelligence","8":"tag-ai","9":"tag-ai-company","10":"tag-anthropic","11":"tag-artificial-intelligence","12":"tag-artificialintelligence","13":"tag-ca","14":"tag-canada","15":"tag-cloudflare","16":"tag-content","17":"tag-heavy-bot-activity","18":"tag-implicit-bargain","19":"tag-information","20":"tag-internet","21":"tag-ratio","22":"tag-referral","23":"tag-technology","24":"tag-time","25":"tag-traffic","26":"tag-user","27":"tag-web","28":"tag-website"},"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/posts\/600496","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/comments?post=600496"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/posts\/600496\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/media\/600497"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/media?parent=600496"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/categories?post=600496"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/tags?post=600496"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}