{"id":44377,"date":"2025-08-05T03:36:13","date_gmt":"2025-08-05T03:36:13","guid":{"rendered":"https:\/\/www.newsbeep.com\/au\/44377\/"},"modified":"2025-08-05T03:36:13","modified_gmt":"2025-08-05T03:36:13","slug":"ai-site-perplexity-uses-stealth-tactics-to-flout-no-crawl-edicts-cloudflare-says","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/au\/44377\/","title":{"rendered":"AI site Perplexity uses \u201cstealth tactics\u201d to flout no-crawl edicts, Cloudflare says"},"content":{"rendered":"

AI search engine Perplexity is using stealth bots and other tactics to evade websites\u2019 no-crawl directives, an allegation that if true violates Internet norms that have been in place for more than three decades, network security and optimization service Cloudflare said Monday.<\/p>\n

In a blog post<\/a>, Cloudflare researchers said the company received complaints from customers who had disallowed Perplexity scraping bots by implementing settings in their sites\u2019 robots.txt files and through Web application firewalls that blocked the declared Perplexity crawlers. Despite those steps, Cloudflare said, Perplexity continued to access the sites\u2019 content.<\/p>\n

The researchers said they then set out to test it for themselves and found that when known Perplexity crawlers encountered blocks from robots.txt files or firewall rules, Perplexity then searched the sites using a stealth bot that followed a range of tactics to mask its activity.<\/p>\n

>10,000 domains and millions of requests<\/p>\n

\u201cThis undeclared crawler utilized multiple IPs not listed in Perplexity\u2019s official IP range, and would rotate through these IPs in response to the restrictive robots.txt policy and block from Cloudflare,” the researchers wrote. \u201cIn addition to rotating IPs, we observed requests coming from different ASNs in attempts to further evade website blocks. This activity was observed across tens of thousands of domains and millions of requests per day.\u201d<\/p>\n

The researchers provided the following diagram to illustrate the flow of the technique they allege Perplexity used.<\/p>\n

<\/p>\n

\"\"<\/p>\n

<\/a><\/p>\n

If true, the evasion flouts Internet norms in place for more than three decades. In 1994, engineer Martijn Koster proposed the Robots Exclusion Protocol<\/a>, which provided a machine-readable format for informing crawlers they weren\u2019t permitted on a given site. Sites that their content indexed installed the simple robots.txt file at the top of their homepage. The standard, which has been widely observed and endorsed ever since, formally became a standard under the Internet Engineering Task Force in 2022<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"AI search engine Perplexity is using stealth bots and other tactics to evade websites\u2019 no-crawl directives, an allegation…\n","protected":false},"author":2,"featured_media":44378,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[256,254,255,64,63,105],"class_list":{"0":"post-44377","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-artificial-intelligence","8":"tag-ai","9":"tag-artificial-intelligence","10":"tag-artificialintelligence","11":"tag-au","12":"tag-australia","13":"tag-technology"},"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/44377","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/comments?post=44377"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/44377\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media\/44378"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media?parent=44377"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/categories?post=44377"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/tags?post=44377"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}