This is where services like the Internet Archive<\/a> and its Wayback Machine come in. Described on the site as a \u201cdigital library of Internet sites and other cultural artifacts in digital form,\u201d the Wayback Machine allows users to look at defunct websites and older versions of current-day sites. This is an invaluable tool<\/a> for researchers, as it allows them to see information that is no longer online in addition to how and when sites and articles have been edited.<\/p>\n However, this tool is about to be slightly less effective, as Reddit<\/a> recently announced that it would be blocking the service from indexing most of the site moving forward. The reason? A.I.<\/p>\n A History of Reddit Limiting Access<\/p>\n As reported by The Verge<\/a>, Reddit will now block the Internet Archive from indexing many of the pages on the site. While the Wayback Machine will still be able to index the homepage, showing which threads on the site were the most popular at a given date and time, they will no longer allow the service to save individual threads.<\/p>\n The reason for this, the social media site says, is the rise of Artificial Intelligence and Large Language Models.\u00a0<\/p>\n In short, while Reddit used to allow free and open access to its API, it has slowly begun to implement fees to use its vast array of content. In 2023, the company announced<\/a> that it would begin charging companies for developer access to its API, and in 2024, it began to charge search engines<\/a> to index its content.<\/p>\n Why the sudden clampdown? Since ChatGPT debuted, there\u2019s been a growing interest in the tech sector about Large Language Models \u2014 and, seeing as Reddit is a massive and constantly updating repository of naturalistic user-generated content in multiple languages, it\u2019s become a great tool for harvesting data to train these LLMs.<\/p>\n Why is Reddit Blocking the Internet Archive from Indexing the Site?<\/p>\n Seeing that LLMs were using Reddit\u2019s data, the site began to charge companies for use, striking a deal with OpenAI <\/a>and Google<\/a> to allow their LLMs to be trained on its data.<\/p>\n The site\u2019s recent clampdown on the Internet Archive<\/a> is claimed to be related to the use of this data. While companies are supposed to pay Reddit to access its broad swath of content, Reddit spokesperson Tim Rathschmidt claims that some companies are circumventing this by downloading the site from saved versions on the Internet Archive.<\/p>\n \u201cInternet Archive provides a service to the open web, but we\u2019ve been made aware of instances where AI companies violate platform policies, including ours, and scrape data from the Wayback Machine,\u201d Rathschmidt told The Verge.<\/p>\n However, this doesn\u2019t appear to be the only reason. Rathschmidt added that \u201cuntil [the Internet Archive is] able to defend their site and comply with platform policies (e.g., respecting user privacy, re: deleting removed content) we\u2019re limiting some of their access to Reddit data to protect redditors.\u201d<\/p>\n These limitations will be implemented slowly, with the company saying that they will \u201cinform [the Internet Archive] of the limits before they go into effect.\u201d In response, Mark Graham, director of the Wayback Machine, said in a statement to The Verge that \u201cWe have a longstanding relationship with Reddit and continue to have ongoing discussions about this matter.\u201d<\/p>\n Redditors React<\/p>\n On Reddit, a thread on the r\/technology subreddit<\/a> about this news quickly racked up over 30 thousand upvotes, with many claiming that stories like these showed how the days of a free and open internet were gradually coming to an end.<\/p>\n \u201cOutrageous, especially with how often posts, threads and users get deleted,\u201d wrote<\/a> a user.<\/p>\n \u201cNew age of internet censorship,\u201d declared a second<\/a>, citing issues like the U.K.\u2019s new age verification law<\/a>.\u00a0<\/p>\n Others questioned whether Reddit was being truthful in their statements, claiming that \u201cscraping\u201d the Internet Archive would be a difficult and time-consuming process. Instead, they alleged other factors may be at play.<\/p>\n \u201cIt\u2019s just bull****. The internet archive has pretty aggressive rate limiting, and the loading speed isn\u2019t very fast in the first place,\u201d said a commenter<\/a>. \u201cScraping the Wayback machine isn\u2019t exactly efficient. It\u2019s just a false pretense to squeeze them for some money.\u201d<\/p>\n \u201cThis makes zero sense. If anyone has used the Internet Archive, they will quickly realize how difficult it would be to scrape because it is so d***ed slow!\u201d exclaimed another<\/a>.<\/p>\n \u201cReddit can\u2019t have people recording all of the admin\/moderator manipulation. It ruins their platform\u2019s credibility. And thus its cultural relevance and shareholder value,\u201d suggested a third<\/a>.<\/p>\n We\u2019ve reached out to Reddit and the Internet Archive via email.<\/p>\n![\"In](\"https:\/\/www.newsbeep.com\/us\/wp-content\/uploads\/2025\/08\/AdobeStock_823032387_Editorial_Use_Only.jpeg\")
Koshiro K\/Adobe Stock<\/p>\n![\"In](\"https:\/\/www.newsbeep.com\/us\/wp-content\/uploads\/2025\/08\/AdobeStock_823030512_Editorial_Use_Only.jpeg\")
Koshiro K\/Adobe Stock<\/p>\n