{"id":162395,"date":"2025-09-17T04:44:06","date_gmt":"2025-09-17T04:44:06","guid":{"rendered":"https:\/\/www.newsbeep.com\/us\/162395\/"},"modified":"2025-09-17T04:44:06","modified_gmt":"2025-09-17T04:44:06","slug":"wtf-is-headless-browsing-and-how-are-ai-agents-fueling-it","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/us\/162395\/","title":{"rendered":"WTF is headless browsing, and how are AI agents fueling it?"},"content":{"rendered":"

Headless browsers \u2014 the behind-the-scenes software that lets machines surf the web like people \u2014 were once the domain of quality-assurance testers and SEO agencies. But new AI-powered browsers launched this last year \u2014 like Perplexity\u2019s Comet<\/a> and Browser Company of New York\u2019s Dia \u2014 are bringing new meaning to the term.<\/p>\n

These players are using headless browsers to power AI agents that need to click, scroll and interact with websites as a human would, to retrieve information.\u00a0<\/p>\n

For media companies, that raises questions: How much traffic is real vs. automated? How should analytics platforms account for agent-driven browsing? And for advertisers, it raises concerns about whether AI-driven sessions will distort measurement or expose new vulnerabilities to fraud.<\/p>\n

Read on to find out what this means for publishers, at a time when many are working to block AI bot traffic<\/a> and protect their content from getting scraped without their control or compensation.<\/p>\n

What is headless browsing in the AI era?<\/p>\n

Here\u2019s how people typically interact with AI engines: A user goes to an AI application (like ChatGPT for example) and types in a prompt. That AI system either operates on its trained knowledge, or goes out to retrieve fresh information from websites (known as retrieval-augmented generation.) These AI bots identify themselves to websites through their user-agent strings, such as ChatGPT-User or PerplexityBot. The AI then synthesizes the retrieved information with its own training to generate the final answer to the user\u2019s prompt.<\/p>\n

But a newer wave of AI interfaces \u2014 including autonomous agents and AI browsers \u2014 works differently. Instead of making targeted retrieval calls and flagging themselves as bots, they rely on headless browsers that actually peruse sites the way a human would: clicking, scrolling and loading pages. Often, these sessions present themselves as standard visitors (for example, using a Chromium string) rather than as a named bot. That makes it much harder for publishers to distinguish automated browsing from real human traffic, according to a new report from Tollbit, a data marketplace for publishers and AI companies, published on Sept. 8.<\/p>\n

The dynamic can be seen in AI web browsers<\/a>. These tools use AI agents that operate inside a browser to visit websites, without the user ever needing to see webpages. And rather than identifying themselves by their user agent name (ChatGPT-User, for example), they present as a standard visitor (such as Chromium, Google\u2019s open-source web browser engine). The result: publishers can\u2019t easily tell what\u2019s human and what\u2019s AI-driven.<\/p>\n

According to Tollbit\u2019s report, AI headless browsers can bypass common bot-blocking techniques. Headless browsers like Browserless claim they can evade these detection tools, and solve CAPTCHAs (which are meant to weed out humans from bots).<\/p>\n

\u201cIn the case of Perplexity Comet, the agent appears to use the user\u2019s own desktop to request webpages when automating workflows. This makes the requests appear to come from a user\u2019s residential IP address, which can confuse [content delivery network] companies into thinking it is legitimate,\u201d Tollbit\u2019s report reads. The report found a 336% increase in websites blocking or redirecting AI bots over the past year.<\/p>\n

How big of a problem are AI headless browsers?<\/p>\n

It\u2019s still early days, but these new AI headless browsers do pose a threat to publishers\u2019 businesses, execs told Digiday.<\/p>\n

The adoption of AI browsers is only going to increase. PayPal and Venmo users (so, millions of people) were just offered access to the Comet browser for free this month<\/a> (previously, it was available to Perplexity subscribers and a small group of early users on a waitlist).<\/p>\n

In addition to new AI browsers on the market, OpenAI is rumored to be developing one<\/a>, and Google is currently developing one through its Project Mariner.\u00a0<\/p>\n

Publishers have already moved to take a stronger stance<\/a> against AI bot traffic and content scraping. AI headless browsers could be the next evolution of that battle.<\/p>\n

\u201cThere\u2019s the notion that whatever work we\u2019re doing here on blocking traffic and crawlers and robots\u2026 that gets circumvented all together,\u201d said an exec at a business publisher, who asked to remain anonymous. But the fact that new challenges are cropping up in this area shows that these efforts are worthwhile, the exec added. \u201cIt\u2019s a reinforcement of, we\u2019re doing the right thing [by] blocking as much as we can.\u201d<\/p>\n

A different exec at a large digital publisher, who requested anonymity, said they believed that, for now, headless browsers posed a minor issue. But it could become a significant problem if major players like Google or OpenAI adopt them.<\/p>\n

What does it mean for publishers?<\/p>\n

It\u2019s the next step in a broader shift: AI systems are starting to replace the need for users to search or visit publisher sites directly, chipping away at publishers\u2019 (human) traffic. TollBit data suggests that human traffic to publisher websites is beginning to decline, while bot traffic is growing. There was a 9.4% decline in the number of human visitors to websites between Q1 and Q2 2025, according to the Tollbit report. Meanwhile, the ratio of site visits coming from AI bots compared to human visitors was 1 in 50, up from 1 in 200 in Q1 2025.<\/p>\n

First of all, this is a transparency issue. Publishers can\u2019t have control over something they can\u2019t fully see or measure. It\u2019s also a business issue. Ads are being served on publishers\u2019 sites to AI agents, rather than humans. According to Toshit Panigrahi, co-founder and CEO of TollBit, ads are being served in OpenAI\u2019s agent mode in ChatGPT, which can complete similar tasks to an AI headless browser.<\/p>\n

\u201cIf this is triggering an actual ad serving call\u2026 multiply that by a million and it can basically be\u2026 unbelievable scale ad fraud,\u201d the business publishing exec said, who added that this has the potential to drive down CPMs.\u00a0<\/p>\n

\u201cThe first [next] step will be information,\u201d the exec said. But it wasn\u2019t like the exec planned to bring this up at the next advertiser client meeting. \u201cAs a publisher that relies on advertising to keep the lights on, we can\u2019t be going out and beating the drum saying, \u2018This is bad.\u2019 We\u2019ll let them come to us on that one.\u201d<\/p>\n

The cost of loading webpages in response to AI agent requests can add up for publishers too, because they have to pay for better cybersecurity tools, Panigrahi said.\u00a0<\/p>\n

Jesse Dwyer, head of communication at Perplexity, said the costs were far greater for Perplexity. \u201cThis does create some real cost considerations for publishers and AI companies to tackle together. When a user sends their Comet assistant to a site, it costs Perplexity 350X to 2300X more than the publisher\u2019s cost to serve that user a single page view. The new costs of giving people a better internet are proof that users have been frustrated by the old internet. So publishers and AI companies are together at the forefront of creating a better, healthier internet,\u201d Dwyer said.<\/p>\n

How can publishers detect headless browsing?<\/p>\n

There are some signals publishers can look for that indicate they\u2019re receiving traffic from AI headless browsers, Panigrahi said.<\/p>\n

If the request isn\u2019t rendering JavaScript, it\u2019s likely not from a human \u2013 because most people have JavaScript in their browsers, Panigrahi said. If the IP address is coming from a data center, that can also be a red flag. The problem is that many of these requests come from residential IP addresses.<\/p>\n

The digital publishing exec said there are a few signs they are looking for now, such as how many people are browsing its sites using the Comet browser. But that still doesn\u2019t make it clear if the browsing is being done by an AI agent or human.<\/p>\n

\u201cAt the moment, there\u2019s no flag getting passed through from the browser to the website that tells you whether the browser is being human-controlled or AI-controlled,\u201d the exec said.<\/p>\n","protected":false},"excerpt":{"rendered":"Headless browsers \u2014 the behind-the-scenes software that lets machines surf the web like people \u2014 were once the…\n","protected":false},"author":2,"featured_media":162396,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[45],"tags":[182,181,507,1873,74,98520],"class_list":{"0":"post-162395","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-artificial-intelligence","8":"tag-ai","9":"tag-artificial-intelligence","10":"tag-artificialintelligence","11":"tag-generative-ai","12":"tag-technology","13":"tag-wtf-series"},"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/posts\/162395","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/comments?post=162395"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/posts\/162395\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/media\/162396"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/media?parent=162395"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/categories?post=162395"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/tags?post=162395"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}