{"id":479525,"date":"2026-06-16T04:55:08","date_gmt":"2026-06-16T04:55:08","guid":{"rendered":"https:\/\/www.newsbeep.com\/nz\/479525\/"},"modified":"2026-06-16T04:55:08","modified_gmt":"2026-06-16T04:55:08","slug":"97-of-llms-txt-files-got-no-requests-ahrefs-data-shows","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/nz\/479525\/","title":{"rendered":"97% Of llms.txt Files Got No Requests, Ahrefs Data Shows"},"content":{"rendered":"<p>Ahrefs analyzed logs from 137,000 domains and found 97% of llms.txt files got zero requests. No bots, no humans.<\/p>\n<p>The <a href=\"https:\/\/ahrefs.com\/blog\/llmstxt-study\/\" target=\"_blank\" rel=\"noopener nofollow\">analysis<\/a> used Ahrefs data to identify user agents fetching files. Around 28% of 137,000 domains publish an llms.txt file, but since Ahrefs\u2019 customers are more technical, the actual adoption on the broader web is likely lower.<\/p>\n<p>Of roughly 38,000 domains with valid files, only about 1,100 received any traffic.<\/p>\n<p>Of files with requests, 96% came from bots, mostly non-AI. AI retrieval bots linked to ChatGPT and Perplexity made up 1%.<\/p>\n<p>Who Fetches llms.txt Files<\/p>\n<p>SEO audit tools had 21% requests, then unidentified bots (14%), web crawlers like Googlebot (13%), and tech profiling tools like BuiltWith (11%).<\/p>\n<p>AI bots, across four categories, made up 19% of requests. AI is the largest segment, but the breakdown differs from most llms.txt advocates\u2019 expectations.<\/p>\n<p>Coding agents sent 10% of requests, training crawlers 5%, assistants 2%. Claude-Code and GPTBot were the top individual bots.<\/p>\n<p>Slackbot alone fetched llms.txt files more often than PerplexityBot did.<\/p>\n<p>The Industry Studying Itself<\/p>\n<p>The report found 12% of requests from tools that audit, scan, or study llms.txt files rather than use them.<\/p>\n<p>GEO and AEO readiness tools sent 5% of requests; dedicated scanners and validators sent 3%, more than AI retrieval bots and assistants combined. Research bots sent 2%, with the largest research crawler identifying as a prompt injection survey.<\/p>\n<p>An ecosystem has developed around scoring and cataloging a file format before a significant audience appears.<\/p>\n<p>No AI Bot Looks For Files That Don\u2019t Exist<\/p>\n<p>Requests for \/llms.txt paths with 404 errors drew no AI traffic. Humans hitting those 404s seem to be people typing the URL into browsers, likely checking competitors.<\/p>\n<p>The Chrome Lighthouse llms.txt audit, which reignited the llms.txt debate in May, generated about 22 requests across the dataset, roughly 1 in 1,000.<\/p>\n<p>Why This Matters<\/p>\n<p>The data lines up with what Google\u2019s John Mueller <a href=\"https:\/\/www.searchenginejournal.com\/googles-mueller-says-llms-txt-cant-help-llms-differentiate-sites\/579304\/\" rel=\"nofollow noopener\" target=\"_blank\">has said<\/a> about llms.txt for over a year. Lily Ray pressed Mueller on the gap between Google Search\u2019s dismissal and Chrome\u2019s Lighthouse audit. He said llms.txt is \u201cnot done for search\u201d and called it a \u201ctemporary crutch, perhaps to save some tokens\u201d for AI coding tools.<\/p>\n<p>The data shows the file\u2019s audience is coding agents and training crawlers, not AI search and retrieval bots that would generate citations.<\/p>\n<p>We <a href=\"https:\/\/www.searchenginejournal.com\/googles-llms-txt-guidance-depends-on-which-product-you-ask\/575431\/\" rel=\"nofollow noopener\" target=\"_blank\">reported<\/a> on the split between Google Search and Lighthouse documentation in May. SE Ranking\u2019s <a href=\"https:\/\/www.searchenginejournal.com\/llms-txt-shows-no-clear-effect-on-ai-citations-based-on-300k-domains\/561542\/\" rel=\"nofollow noopener\" target=\"_blank\">earlier analysis<\/a> of 300,000 domains showed no connection between having llms.txt and AI citation frequency. Ahrefs\u2019 data points to one possible reason: the bots most directly tied to live AI retrieval barely requested these files in May.<\/p>\n<p>Looking Ahead<\/p>\n<p>The prompt injection finding is worth watching. Ahrefs found a crawler studying llms.txt as a prompt injection risk, since agents trust ingested content. Sites auto-generating these files via CMS should review their content.<\/p>\n<p>Every figure in this report is a ceiling. Ahrefs measured requests, not whether bots acted on what they fetched.<\/p>\n<p>Featured Image: sdecoret\/Shutterstock<\/p>\n","protected":false},"excerpt":{"rendered":"Ahrefs analyzed logs from 137,000 domains and found 97% of llms.txt files got zero requests. No bots, no&hellip;\n","protected":false},"author":2,"featured_media":479526,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[111,139,69,145],"class_list":["post-479525","post","type-post","status-publish","format-standard","has-post-thumbnail","category-technology","tag-new-zealand","tag-newzealand","tag-nz","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts\/479525","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/comments?post=479525"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts\/479525\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/media\/479526"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/media?parent=479525"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/categories?post=479525"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/tags?post=479525"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}