{"id":54325,"date":"2025-08-09T02:01:08","date_gmt":"2025-08-09T02:01:08","guid":{"rendered":"https:\/\/www.newsbeep.com\/au\/54325\/"},"modified":"2025-08-09T02:01:08","modified_gmt":"2025-08-09T02:01:08","slug":"the-war-for-the-web-has-begun","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/au\/54325\/","title":{"rendered":"The War for the Web Has Begun"},"content":{"rendered":"

A high-stakes war has just broken out over the future of the internet. In one corner is Cloudflare, a giant of web infrastructure that acts as a gatekeeper for a huge portion of online traffic. In the other is Perplexity, a darling of the AI world, a search engine threatening to upend Google\u2019s dominance.<\/p>\n

The accusation is explosive: Cloudflare claims Perplexity is a bad actor, a rogue bot that ignores the internet\u2019s oldest rules to secretly scrape data from websites that have explicitly told it to stay away. Perplexity\u2019s response is just as fiery: it says Cloudflare is either dangerously incompetent or engaged in a publicity stunt, fundamentally misunderstanding how modern AI works.<\/p>\n

The feud is the first major battle in a conflict that will define the next era of the web: Who gets to access online information, and who gets to decide the rules?<\/p>\n

The Accusation: A Rogue Bot in Disguise <\/p>\n

For decades, the internet has operated on a \u201cgentleman\u2019s agreement\u201d called the robots.txt file. It\u2019s a simple text file that website owners use to post a digital \u201cDo Not Enter\u201d sign for automated web crawlers or \u201cbots.\u201d Well-behaved bots, like Google\u2019s, respect this sign.<\/p>\n

In a scathing blog post<\/a>, Cloudflare alleges that Perplexity is ignoring it. The company claims that when its declared bot, \u201cPerplexityBot,\u201d is blocked, the AI search engine switches to stealth mode, using generic browser identities and rotating IP addresses to continue crawling and gathering data in disguise.<\/p>\n

Cloudflare says it tested this by creating brand-new, private websites with strict \u201cno bots allowed\u201d rules. Despite this, they found that \u201cPerplexity was still providing detailed information regarding the exact content hosted on each of these restricted domains.\u201d Based on this \u201cstealth crawling behavior,\u201d Cloudflare announced it has now de-listed Perplexity as a verified bot and is actively blocking its undeclared crawlers.<\/p>\n

The Rebuttal: \u201cYou Don\u2019t Understand How AI Works\u201d <\/p>\n

Perplexity\u2019s response<\/a> was swift, accusing Cloudflare of getting \u201calmost everything wrong about how modern AI assistants actually work.\u201d The company argues that it is not a traditional \u201cbot\u201d and that Cloudflare is misapplying old rules to new technology.<\/p>\n

The core of their argument is the difference between a bot and a user agent. A traditional bot, like Google\u2019s, systematically crawls billions of pages to build a massive index for later use. A user agent, Perplexity claims, acts on behalf of a real person in real-time. When you ask Perplexity a question, its AI agent fetches the necessary information from the web at that moment to answer you. It\u2019s not stockpiling data; it\u2019s acting as your personal research assistant.<\/p>\n

\u201cThis is fundamentally different from traditional web crawling in which crawlers systematically visit millions of pages to build massive databases, whether anyone asked for that specific information or not,\u201d Perplexity wrote in a detailed response. \u201cWhen companies like Cloudflare mischaracterize user-driven AI assistants as malicious bots, they\u2019re arguing that any automated tool serving users should be suspect\u2014a position that would criminalize email clients and web browsers.\u201d<\/p>\n

Then came the bombshell counter-accusation. Perplexity claims Cloudflare \u201cfundamentally misattributed 3-6M daily requests\u201d from a third-party cloud browser service to Perplexity, calling it a \u201cbasic traffic analysis failure that\u2019s particularly embarrassing for a company whose core business is understanding and categorizing web traffic.\u201d Perplexity suggests this is either a \u201cclever publicity moment\u201d or a sign that Cloudflare is \u201cdangerously misinformed on the basics of AI.\u201d<\/p>\n

Users on social media were divided. \u201cPerplexity is just using a proxy to fetch something that\u2019s already on the public web, to answer a user\u2019s question. Framing it as some kind of attack is absurd. The public web should be public,\u201d defended tech founder Andrej Radonjic. Another user was more critical: \u201cPerplexity, pretending to be a search engine, pretending to be AI, yet neither.\u201d<\/p>\n

perplexity is just using a proxy to fetch something that\u2019s already on the public web, to answer a user\u2019s question.<\/p>\n

framing it as some kind of attack is absurd. the public web should be public.<\/p>\n

\u2014 Andrej (@0xdrej) August 4, 2025<\/a><\/p>\n

Who Owns the Open Web? <\/p>\n

This public feud lays bare the central tension of the AI era. AI startups like Perplexity need access to the vast ocean of data on the open web to function and compete with giants like Google and OpenAI. Without it, they can\u2019t provide real-time, accurate answers. But website owners are growing increasingly wary of having their content scraped without consent or compensation to train and power these new AI models.<\/p>\n

Cloudflare, by choosing to block Perplexity\u2019s undeclared crawlers, has effectively appointed itself as the AI data police, making decisions about what constitutes \u201clegitimate\u201d web traffic. Perplexity warns this could lead to a \u201ctwo-tiered internet\u201d where access depends not on a user\u2019s needs, but on whether their chosen AI tool has been \u201cblessed by infrastructure controllers.\u201d<\/p>\n

The rules of the internet are being rewritten in real-time. The old gentleman\u2019s agreement is breaking down, and the battle between the gatekeepers and the innovators has just begun. The outcome will determine not just the future of AI, but the future of the open web itself.<\/p>\n