{"id":350354,"date":"2025-12-15T21:43:06","date_gmt":"2025-12-15T21:43:06","guid":{"rendered":"https:\/\/www.newsbeep.com\/au\/350354\/"},"modified":"2025-12-15T21:43:06","modified_gmt":"2025-12-15T21:43:06","slug":"googlebot-tops-ai-crawler-traffic","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/au\/350354\/","title":{"rendered":"Googlebot Tops AI Crawler Traffic"},"content":{"rendered":"

Cloudflare published its sixth annual Year in Review<\/a>, offering a comprehensive looks at Internet traffic, security, and AI crawler activity across 2025.<\/p>\n

The report draws on data from Cloudflare\u2019s network, which spans more than 330 cities across 125 countries and handles over 81 million HTTP requests per second on average.<\/p>\n

The AI crawler findings stand out. Googlebot crawled far more web pages than any other AI bot, reflecting Google\u2019s dual-purpose approach to crawling for both search indexing and AI training.<\/p>\n

Googlebot Top AI Crawler Traffic<\/p>\n

Cloudflare analyzed successful requests for HTML content from leading AI crawlers during October and November 2025. The results showed Googlebot reached 11.6% of unique web pages in the sample.<\/p>\n

That\u2019s more than 3 times the pages seen by OpenAI\u2019s GPTBot at 3.6%. It\u2019s nearly 200 times more than PerplexityBot, which crawled just 0.06% of pages.<\/p>\n

Bingbot came in third at 2.6%, followed by Meta-ExternalAgent<\/a> and ClaudeBot<\/a> at 2.4% each.<\/p>\n

The report noted that because Googlebot crawls for both search indexing and AI model training, web publishers face a difficult choice. Blocking Googlebot\u2019s AI training means risking search discoverability.<\/p>\n

Cloudflare wrote:<\/p>\n

\u201cBecause Googlebot is used to crawl content for both search indexing and AI model training, and because of Google\u2019s long-established dominance in search, Web site operators are essentially unable to block Googlebot\u2019s AI training without risking search discoverability.\u201d<\/p>\n

Related: Complete Crawler List For AI User-Agents<\/a><\/p>\n

AI Bots Now Account For 4.2% of HTML Requests<\/p>\n

Throughout 2025, AI bots (excluding Googlebot) averaged 4.2% of HTML requests across Cloudflare\u2019s customer base. The share fluctuated between 2.4% in early April and 6.4% in late June.<\/p>\n

Googlebot alone accounted for 4.5% of HTML requests, slightly more than all other AI bots combined.<\/p>\n

The share of human-generated HTML traffic started 2025 at seven percentage points below non-AI bot traffic. By September, human traffic began exceeding non-AI bot traffic on some days. As of December 2, humans generated 47% of HTML requests while non-AI bots generated 44%.<\/p>\n

Crawl-to-Refer Ratios Show Wide Variation<\/p>\n

Cloudflare tracks how often AI and search platforms send traffic to sites relative to how often they crawl. A high ratio means heavy crawling without sending users back to source sites.<\/p>\n

Anthropic had the highest ratios among AI platforms, ranging from approximately 25,000:1 to 100,000:1 during the second half of the year after stabilizing from earlier volatility.<\/p>\n

OpenAI\u2019s ratios reached as high as 3,700:1 in March. Perplexity maintained the lowest ratios among leading AI platforms, generally below 400:1 and under 200:1 from September onward.<\/p>\n

For comparison, Google\u2019s search crawl-to-refer ratio stayed much lower, generally between 3:1 and 30:1 throughout the year.<\/p>\n

User-Action Crawling Grew Over 20X<\/p>\n

Not all AI crawling is for model training. \u201cUser action\u201d crawling occurs when bots visit sites in response to user questions posed to chatbots.<\/p>\n

This category saw the fastest growth in 2025. User-action crawling volume increased more than 15 times from January through early December. The trend closely matched the traffic pattern for OpenAI\u2019s ChatGPT-User bot<\/a>, which visits pages when users ask ChatGPT questions.<\/p>\n

The growth showed a weekly usage pattern starting in mid-February, suggesting increased use in schools and workplaces. Activity dropped during June through August when students were on break and professionals took vacations.<\/p>\n

AI Crawlers Most Blocked In Robots.txt<\/p>\n

Cloudflare analyzed robots.txt files<\/a> across nearly 3,900 of the top 10,000 domains. AI crawlers were the most frequently blocked user agents.<\/p>\n

GPTBot, ClaudeBot, and CCBot had the highest number of full disallow directives. These directives tell crawlers to stay away from entire sites.<\/p>\n

Googlebot and Bingbot showed a different pattern. Their disallow directives leaned heavily toward partial blocks, likely focused on login endpoints and non-content areas rather than full site blocking.<\/p>\n

Civil Society Became Most-Attacked Sector<\/p>\n

For the first time, organizations in the \u201cPeople and Society\u201d vertical were the most targeted by attacks. This category includes religious institutions, nonprofits, civic organizations, and libraries.<\/p>\n

The sector received 4.4% of global mitigated traffic, up from under 2% at the start of the year. Attack share jumped to over 17% in late March and peaked at 23.2% in early July.<\/p>\n

Many of these organizations are protected by Cloudflare\u2019s Project Galileo.<\/p>\n

Gambling and games, the most-attacked vertical in 2024, saw its share drop by more than half to 2.6%.<\/p>\n

Other Key Findings<\/p>\n

Cloudflare\u2019s report included several additional findings across traffic, security, and connectivity.<\/p>\n

Global Internet traffic grew 19% year-over-year. Growth stayed relatively flat through mid-April, then accelerated after mid-August.<\/p>\n

Post-quantum encryption now secures 52% of human traffic to Cloudflare, nearly double the 29% share at the start of the year.<\/p>\n

ChatGPT remained the top generative AI service globally. Google Gemini, Windsurf AI, Grok\/xAI, and DeepSeek were new entrants to the top 10.<\/p>\n

Starlink traffic doubled in 2025, with service launching in more than 20 new countries.<\/p>\n

Nearly half of the 174 major Internet outages observed globally were caused by government-directed shutdowns. Cable cut outages dropped nearly 50%, while power failure outages doubled.<\/p>\n

European countries dominated Internet quality metrics. Spain topped the list for overall Internet quality, with average download speeds above 300 Mbps.<\/p>\n

Why This Matters<\/p>\n

The AI crawler data should affects how you think about bot access and traffic.<\/p>\n

Google\u2019s dual-purpose crawler creates a competitive advantage. You can block other AI crawlers while keeping Googlebot access for search visibility, but you can\u2019t separate Google\u2019s search crawling from its AI training crawling.<\/p>\n

The crawl-to-refer ratios help quantify what publishers already suspected. AI platforms crawl heavily but send little traffic back. The gap between crawling and referring varies widely by platform.<\/p>\n

The civil society attack data matters if you work with nonprofits or advocacy organizations. These groups now face the highest rate of attacks.<\/p>\n

Looking Ahead<\/p>\n

Cloudflare expects AI metrics to change as the space continues to evolve. The company added several new AI-related datasets to this year\u2019s report that weren\u2019t available in previous editions.<\/p>\n

The crawl-to-refer ratios may change as AI platforms adjust their search features and referral behavior. OpenAI\u2019s ratios already showed some decline through the year as ChatGPT search usage grew.<\/p>\n

For robots.txt management, the data shows most publishers are choosing partial blocks for major search crawlers while fully blocking AI-only crawlers. The year-end state of these directives provides a baseline for tracking how publisher policies evolve in 2026.<\/p>\n

Featured Image: Mamun_Sheikh\/Shutterstock<\/p>\n","protected":false},"excerpt":{"rendered":"Cloudflare published its sixth annual Year in Review, offering a comprehensive looks at Internet traffic, security, and AI…\n","protected":false},"author":2,"featured_media":350355,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[256,254,255,64,63,105],"class_list":{"0":"post-350354","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-artificial-intelligence","8":"tag-ai","9":"tag-artificial-intelligence","10":"tag-artificialintelligence","11":"tag-au","12":"tag-australia","13":"tag-technology"},"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/350354","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/comments?post=350354"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/350354\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media\/350355"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media?parent=350354"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/categories?post=350354"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/tags?post=350354"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}