![\"Robot](\"https:\/\/www.newsbeep.com\/uk\/wp-content\/uploads\/2026\/01\/shutterstock_2680036993-e1769012550733-1038x778.webp.webp\")
\n Picture: Shutterstock\/Ketmut<\/p>\n
\n Picture: Shutterstock\/Ketmut<\/p>\n
Almost eight in ten of the biggest news websites in the UK and US are now blocking AI<\/a> training bots.<\/p>\n This week in London publishers were told it is \u201cnever too late\u201d to begin blocking the bots due to the frequency with which new crawling needs to take place via retrieval-augmented generation so that AI chatbot responses remain current.<\/p>\n Some 79% of almost 100 top news websites in the UK and US are blocking at least one crawler used for AI training out of OpenAI\u2019s GPTBot, ClaudeBot, Anthropic-ai, CCBot, Applebot-Extended and Google-Extended.<\/p>\n Meanwhile 71% are blocking AI bots from crawling their sites for retrieval or live searches. Those bots are: ChatGPT-User, Claude-Web, Perplexity-User and OAI-SearchBot.<\/p>\n The latest analysis of who was blocking what was carried out by digital PR platform Buzzstream, looking at a combined and deduplicated list of the 50 biggest news websites in each of the UK<\/a> and US<\/a>, and shared with Press Gazette.<\/p>\n Among the 50 biggest, the publishers that had blocked all the AI bots included in the analysis were: the BBC (both its .co.uk<\/a> and .com domains), The New York Times, the Daily Mail, The Telegraph, Sky News, AP News, New York Post, Newsweek, NBC News, Wall Street Journal, Metro, Business Insider, ABC News, Buzzfeed, Huffpost and The Hill. That means 34% of the top 50 blocked all the bots.<\/p>\n Some 14% of the top-50 publishers allowed access to all 11 AI crawlers analysed: Fox News, The Independent, GB News, Substack, the Standard, the Drudge Report and Politico.<\/p>\n Telegraph SEO director Harry Clarkson-Bennett told Buzzstream: \u201cPublishers are blocking AI bots using the robots.txt because there\u2019s almost no value exchange.<\/a><\/p>\n \u201cLLMs are not designed to send referral traffic and publishers (still) need traffic to survive.<\/p>\n \u201cSo most of us block AI bots because these companies are not willing to pay for the content their model has been trained on and their output is almost entirely internal.\u201d<\/p>\n Across the 50 biggest sites analysed, Perplexity-User was the most permitted (by 32) followed by ChatGPT-User (24). Both are retrieval\/live search crawlers that determine whether Perplexity or ChatGPT can access a site in real time to provide answers to user questions. Conversely, Perplexity-Bot surfaces and links to websites within search results on Perplexity.<\/p>\n The least permitted was Anthropic\u2019s training bot (eight) followed by the Common Crawl web archive bot CCBot (nine).<\/p>\n In February 2024 Press Gazette analysis showed that 61 out of the 106 biggest news websites in the UK and US were blocking at least one AI crawler<\/a>, showing an increase in the past two years.<\/p>\n Google Extended \u201callows you to say yes to search, no to AI training\u201d, Google\u2019s senior manager for government affairs and public policy Roxanne Carter told UK Parliament last week.<\/a><\/p>\n Google Extended lets publishers opt out from AI chatbot Gemini and AI development platform Vertex scraping their content, but does not stop sites from being accessed and used in Google\u2019s AI Overviews.<\/p>\n To avoid this publishers would have to opt out of being scraped by Googlebot, which indexes for search. Googlebot was not included in the new Buzzstream analysis but opting out of it has been described as an \u201cunenviable choice\u201d.<\/a><\/p>\n Buzzstream found that US publishers are \u201cmuch more aggressive\u201d in blocking Google Extended (58%) than UK publishers (29%).<\/p>\n Among the 100 top news sites in the US and UK, American sites are stricter on training bots (81% blocking) than their UK colleagues (77%).<\/p>\n Robots.txt is not a foolproof way of blocking AI bots as their directives can be ignored by AI companies or circumvented in ways such as using third-party companies to scrape content instead.<\/a><\/p>\n Typos and misconfigurations may stop publishers blocking effectively<\/p>\n Anthony Katsur, CEO of IAB Tech Lab, told the Interactive Advertising Bureau (IAB)\u2019s Techtonic event in London on Wednesday that they had \u201cscoured a large percentage\u201d of publisher commands to AI bots and found \u201cthere\u2019s a lot of typos or a lot of misconfigurations of robots.txt, which means that may not be obeyed \u2013 not that it\u2019s always obeyed anyway\u201d.<\/p>\n Katsur recommended publishers block AI bots, even if they have not yet, because of retrieval-augmented generation which means generative AI models retrieve and reference new information from the web in real time (rather than relying on one initial discovery crawl).<\/p>\n Katsur said: \u201cPublishers, if you have not blocked and you think that it is too late to block, you are wrong. It is never too late to block.<\/p>\n \u201cThe LLMs will come back and they will recrawl content in order for that information to stay fresh and relevant and accurate.\u201d<\/p>\n He also said: \u201cIn the history of mankind, no marketplace has ever been created when things are given away for free or just stolen. So without scarcity, marketplaces won\u2019t exist. So publishers, if you are not blocking, then it is unlikely that some sort of content marketplace will take shape.\u201d<\/p>\n Katsur suggested that publishers should collectively \u201clink arms\u201d and block the bots \u201cfull stop\u201d, even if only for 72 hours, \u201cjust to show that there\u2019s strength amongst publishers, big and small\u201d.<\/p>\n He said that if publishers were threatened with an antitrust or collusion lawsuit as a result, \u201cthat would be a good problem to have because it would mean you\u2019re still in business\u201d.<\/p>\n Katsur said the IAB Tech Lab believes the \u201cdays of crawling\u201d are not \u201csustainable for publishers\u201d.<\/p>\n \u201cWe also don\u2019t necessarily think it\u2019s sustainable for LLMs\u2026 as we see the AI bubble burst, I think from a fiscal or a funding perspective, easy access to ongoing purchase of GPUs [graphics processing units] to fund the LLMs will become constrained, therefore you will need greater efficiencies in terms of how content is accessed and structured for more efficient LLMs.\u201d<\/p>\n![\"Anthony](\"https:\/\/www.newsbeep.com\/uk\/wp-content\/uploads\/2026\/01\/iab_techtonic_2026_021-800x583.webp.webp\")
Anthony Katsur, CEO, IAB Tech Lab speaking at IAB UK Techtonic in London on Wednesday 21 January 2026. Picture: Ben Stevens Photography\/IAB UK<\/p>\n