{"id":459658,"date":"2026-02-05T08:10:14","date_gmt":"2026-02-05T08:10:14","guid":{"rendered":"https:\/\/www.newsbeep.com\/au\/459658\/"},"modified":"2026-02-05T08:10:14","modified_gmt":"2026-02-05T08:10:14","slug":"news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/au\/459658\/","title":{"rendered":"News sites are locking out the Internet Archive to stop AI crawling. Is the \u2018open web\u2019 closing?"},"content":{"rendered":"

When the World Wide Web went live in the early 1990s, its founders hoped<\/a> it would be a space for anyone to share information and collaborate. But today, the free and open web is shrinking.<\/p>\n

The Internet Archive<\/a> has been recording the history of the internet and making it available to the public through its Wayback Machine<\/a> since 1996<\/a>. Now, some of the world\u2019s biggest news outlets are blocking<\/a> the archive\u2019s access to their pages.<\/p>\n

Major publishers \u2013 including The Guardian, The New York Times, the Financial Times, and USA Today \u2013 have confirmed they\u2019re ending the Internet Archive\u2019s access to their content. <\/p>\n

While publishers say<\/a> they support the archive\u2019s preservation mission, they argue unrestricted access creates unintended consequences, exposing journalism to AI crawlers and members of the public trying to skirt their paywalls. <\/p>\n

Yet, publishers don\u2019t simply want to lock out AI crawlers. Rather, they want to sell their content<\/a> to data-hungry tech companies. Their back catalogues of news, books and other media have become a hot commodity<\/a> as data to train AI systems.<\/p>\n

Robot readers<\/p>\n

Generative AI systems such as ChatGPT, Copilot and Gemini require access to large archives of content (such as media content, books, art and academic research) for training<\/a> and to answer user prompts<\/a>. <\/p>\n

Publishers claim technology companies have accessed a lot of this content for free and without the consent of copyright owners<\/a>. Some began taking tech companies to court, claiming they had stolen their intellectual property. High-profile examples include The New York Times<\/a>\u2019 case against ChatGPT\u2019s parent company OpenAI and News Corp\u2019s lawsuit against Perplexity AI<\/a>. <\/p>\n

\"The<\/a><\/p>\n

The New York Times has sued OpenAI for alleged copyright infringement.
\n Sarah Yenesel\/EPA<\/a><\/p>\n

Old news, new money<\/p>\n

In response, some tech companies have struck<\/a> deals<\/a> to pay for access to publishers\u2019 content. NewsCorp\u2019s contract with OpenAI is reportedly worth more than US$250 million<\/a> over five years.<\/p>\n

Similar deals have been struck between academic publishers and tech companies. Publishing houses such as Taylor & Francis and Elsevier<\/a> have come under scrutiny in the past for locking publicly funded research behind commercial paywalls. <\/p>\n

Now, Taylor & Francis<\/a> has signed a US$10 million nonexclusive deal with Microsoft granting the company access to over 3,000 journals. <\/p>\n

Publishers are also using technology to stop unwanted AI bots<\/a> accessing their content, including the crawlers used by the Internet Archive to record internet history. News publishers have referred to the Internet Archive as a \u201cback door<\/a>\u201d to their catalogues, allowing unscrupulous tech companies to continue scraping their content.<\/p>\n

\"A<\/a><\/p>\n

The Internet Archive has been systematically archiving the web for about three decades.
\n Serene Lee\/SOPA Images\/LightRocket via Getty Images<\/a><\/p>\n

The cost of making news free<\/p>\n

The Wayback Machine has also been used by members of the public to avoid newspaper paywalls<\/a>. Understandably, media outlets want readers to pay for news. <\/p>\n

News is a business, and its advertising revenue model<\/a> has come under increasing pressure from the same tech companies using news content for AI training and retrieval. But this comes at the expense of public access to credible information. <\/p>\n

When newspapers first started moving their content online and making it free to the public in the late 1990s, they contributed to the ethos of sharing and collaboration on the early web.<\/p>\n

In hindsight, however, one commentator called free access the \u201coriginal sin<\/a>\u201d of online news. The public became accustomed to getting their digital editions for free, and as online business models shifted, many mid- and small-sized news companies struggled to fund their operations.<\/p>\n

The opposite approach \u2013 placing all commercial news behind paywalls \u2013 has its own problems. As news publishers move to subscription-only models<\/a>, people have to juggle multiple expensive subscriptions or limit their news appetite<\/a>. Otherwise, they\u2019re left with whatever news remains online for free or is served up by social media algorithms<\/a>. The result is a more closed, commercial internet.<\/p>\n

This isn\u2019t the first time that the Internet Archive has been in the crosshairs of publishers<\/a>, as the organisation was previously sued and found to be in breach of copyright through its Open Library project.<\/p>\n

The past and future of the internet<\/p>\n

The Wayback Machine has served as a public record of the web for more than three decades<\/a>, used by researchers, educators, journalists and amateur internet historians. <\/p>\n

Blocking its access to international newspapers of note will leave significant holes in the public record of the internet. <\/p>\n

Today, you can use the Wayback Machine<\/a> to see The New York Times\u2019 front page from June 1997: the first time the Internet Archive crawled the newspaper\u2019s website. In another 30 years, internet researchers and curious members of the public won\u2019t have access to today\u2019s front page, even if the Internet Archive is still around.<\/p>\n

Today\u2019s websites become tomorrow\u2019s historical records. Without the preservation efforts of not-for-profit organisations like The Internet Archive, we risk losing vital records<\/a>.<\/p>\n

Despite the actions of commercial publishers and emerging challenges of AI<\/a>, not-for-profit organisations such as the Internet Archive and Wikipedia<\/a> aim to keep the dream of an open, collaborative and transparent internet alive.<\/p>\n","protected":false},"excerpt":{"rendered":"When the World Wide Web went live in the early 1990s, its founders hoped it would be a…\n","protected":false},"author":2,"featured_media":459659,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[256,254,255,64,63,105],"class_list":{"0":"post-459658","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-artificial-intelligence","8":"tag-ai","9":"tag-artificial-intelligence","10":"tag-artificialintelligence","11":"tag-au","12":"tag-australia","13":"tag-technology"},"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/459658","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/comments?post=459658"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/459658\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media\/459659"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media?parent=459658"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/categories?post=459658"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/tags?post=459658"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}