The head of the IAB Tech Lab has urged Australian media owners to join forces and collectively block AI models from scraping their content, likening the current dynamic to: “Why buy the cow when you can get the milk for free?”

Anthony Katsur, the New York–based CEO of the IAB Tech Lab, said large language models are already wiping out up to 60% of US publishers’ referral traffic, an “unsustainable” trajectory for the media industry.

Speaking to Mumbrella during an IAB trip to Sydney, Katsur said LLMs perform “anywhere from 250 to several thousand” crawls before delivering a single site visit, compared with roughly six crawls per visit from Google Search.

“There’s still a debate amongst some publishers on whether or not to block [LLM crawlers]. I would advise any publisher … you should block. No market has ever been created when people give things away for free.”

ADVERTISEMENT

In terms of the market, IAB Australia’s latest annual report found online advertising expenditure reached $17.2 billion in FY25, a 10% increase, although local media investment continues to lag.

According to the IAB, digital advertising makes up nearly three-quarters of the paid ad market, with search driving roughly 44% of that investment.

In light of this, Katsur is in Sydney meeting media owners to explore “public policy or publisher coalitions” that could help block traffic losses to LLMs.

The current situation carries notable echoes of the mid-2010s, when publishers relied heavily on referral traffic from social platforms, particularly Facebook, to drive audience growth.

That model collapsed when Facebook abruptly shifted its algorithm to de-prioritise news, triggering a sharp drop in traffic and ultimately prompting government intervention.

This paved the way for the development of the News Media Bargaining Code, forcing platforms and publishers into a series of negotiated deals, of which the latest proposal could see platforms with revenues of at least $250 million face a levy on revenue of perhaps 2.25%.

Katsur said he’s seeing the early signs of a similar tension emerging around generative AI, with Australian publishers divided on how aggressively to push back.

IAB Tech Lab’s Anthony Katsur

“[In Australia] it’s a blend,” he said. “Some licensing agreements have already been cut. Other publishers are still plotting out their strategy for how they want to approach the LLMs, which isn’t too dissimilar to what I’m seeing in the US.”

Enjoying Mumbrella? Sign up for our free daily newsletter.

On the technical front, Katsur described blocking measures as a “bit of cat and mouse,” but stressed they are not hugely difficult. Major edge-compute platforms such as Fastly, Akamai, Cloudflare and Cloudfront, along with tools like Datadome, can block known and unnamed LLM crawlers.

For Katsur, the challenge is more business than technical.

Publishers will need to rethink metrics and monetisation models for AI-era content. He warned that many current approaches, such as charging per crawl, “don’t scale and aren’t viable long-term.”

According to Katsur, this structural approach will reflect a publisher’s contractual agreement with any LLM, while tracking “how that content is accessed, how that content is tiered from a pricing perspective and, most importantly, how publishers can measure how that content is used and how it manifests itself.”

He said the framework will apply both to what he terms a “classic chat response,” such as ChatGPT, as well as new browsers like OpenAI’s Atlas.

Despite the rise of LLMs, Katsur remains somewhat optimistic, noting that a “healthy scepticism” will keep consumers returning to the media source.

“If you are a quality media company that has scale, I think it’s going to be a rocky road, but I think it’ll be fun,” he added.