{"id":467539,"date":"2026-06-08T23:40:13","date_gmt":"2026-06-08T23:40:13","guid":{"rendered":"https:\/\/www.newsbeep.com\/nz\/467539\/"},"modified":"2026-06-08T23:40:13","modified_gmt":"2026-06-08T23:40:13","slug":"news-sites-are-blocking-internet-archive-over-ai-scraping-fears","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/nz\/467539\/","title":{"rendered":"News Sites Are Blocking Internet Archive Over AI Scraping Fears"},"content":{"rendered":"

Especially in this era of the Internet, the role of the Internet Archive\u2019s Wayback Machine has become increasingly essential as more and more web content vanishes into the ether or is surreptitiously altered to hide salient details. More recently a new worry has seemingly cropped up in the form of scraping of data for so-called AI systems, or at least that\u2019s part of the excuses being offered for blocking the Wayback Machine\u2019s web crawlers, with [Andrew Deck] and [Hanaa\u2019 Tameez] of [Nieman Lab] detailing the impact and reasons provided<\/a>.<\/p>\n

Some news outlets like The Baltimore Banner insist that they\u2019re only blocking the Wayback Machine crawlers because they are worried that LLM chatbots would otherwise \u2018improperly cite\u2019 the source of content, while outlets like The Atlantic have put a blanket anti-scraping policy in place. Meanwhile news outlets are generally happy to let paid commercial news archiving outlets like ProQuest and LexisNexis index their content, showing a potential financial incentive.<\/p>\n

Whatever the reasons, the direct effect is that as content is modified or vanishes during for example a system migration, buy-out or bankruptcy, researchers who rely on the Wayback Machine are pretty much forced to rely on paid offerings by ProQuest and kin, without the pure archiving focus and free access to information. It will also leave big holes in what the Wayback Machine can cover in its archives, with news especially becoming very spotty.<\/p>\n

Incidentally there\u2019s an ongoing petition over at SaveTheArchive.com<\/a> which people can sign.<\/p>\n","protected":false},"excerpt":{"rendered":"Especially in this era of the Internet, the role of the Internet Archive\u2019s Wayback Machine has become increasingly…\n","protected":false},"author":2,"featured_media":467540,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[365,363,364,111,139,69,145],"class_list":["post-467539","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-new-zealand","tag-newzealand","tag-nz","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts\/467539","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/comments?post=467539"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts\/467539\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/media\/467540"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/media?parent=467539"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/categories?post=467539"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/tags?post=467539"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}