Whatever you think of that, Google and AI companies have for the last few years benefited massively, treating the internet like a free buffet by scraping information off websites and condensing it into pithy answers \u2014 before anyone could say otherwise.<\/p>\n
The end result, which is probably here to stay, is a shift in how we use the internet, likely more convenient for the user. But what\u2019s the cost to the people who feed the machine? How are writers, artists and journalists footing the bill of this new paradigm, and can AI companies be made to pay their fair share?\u00a0<\/p>\n
Take a newspaper like The Park Record, for instance. Original reporting, while still valuable as print media, has also become fodder for the aggregate machine of the online world. That\u2019s a change to the internet\u2019s longtime business model, one which Google had originally helped facilitate: a search engine points to a website, and the website can sell ads or charge a subscription \u2014 or not. That was the deal.\u00a0<\/p>\n
![\"\"](\"https:\/\/www.newsbeep.com\/au\/wp-content\/uploads\/2025\/09\/cloudflare-logo.png\"){kind=logo}
Credit: Cloudflare<\/p>\n
As that dynamic has changed, Cloudflare is one company seeking to rebalance content creators\u2019 leverage in the AI era.\u00a0<\/p>\n
The California-based company works with customers to build and run applications \u201cso they\u2019re fast and secure,\u201d Cohen said, claiming that more than 20% of the internet is built with Cloudflare. Small companies, big companies, free customers and paying customers, a lot of people rely on Cloudflare, including The Park Record. After all, Cloudflare\u2019s CEO and co-founder is Matthew Prince, co-owner of The Park Record with his wife, Tatiana.\u00a0<\/p>\n
Cloudflare\u2019s services are the kinds of things that will make sense to you if you\u2019re computer savvy, and sound like nonsense otherwise: content delivery networks, web access filters, DDoS mitigation \u2014 stuff that\u2019s important to the internet running smoothly.\u00a0<\/p>\n
For talking about AI, the company\u2019s bot management services are most important, said Cohen.<\/p>\n
\u201cBasically, (we) make sure good traffic gets through as fast and securely as possible,\u201d she explained, \u201cand we deflect bad traffic, traffic you don\u2019t want on your site. We use that same network to protect employees \u2026 your data, your network on the internet.\u201d<\/p>\n
Good in, bad out. Around when Google started implementing AI Overview, Cloudflare started hearing from its customers.\u00a0<\/p>\n
\u201cThe way that traffic was coming to (customers\u2019 sites) was changing, and my guess is that\u2019s true for The Park Record; it was true for publishers and content creators of all sizes,\u201d Cohen said.<\/p>\n
They dug into the data and found that what customers were saying was exactly right, Cohen said. AI was eating up traffic.\u00a0<\/p>\n
\u201cAnd if you thought about it and you thought about your own personal life, what was going on is that you\u2019re reading the derivative rather than the original,\u201d she said. \u201cBut you still need the original source to have a thriving internet, to have an internet that helps you answer these types of questions. And so we realized that this was going to increasingly be a problem.\u201d<\/p>\n
More specifically, Prince, on a panel at the recent Fortune Brainstorm Tech 2025 \u2014 a tech-focused conference that Fortune Live Media on Sept. 8-10 at the Montage Deer Valley \u2014 said that the AI Overview feature has made it \u201c10 times harder to get traffic from Google than it was just a few years ago.\u201d\u00a0<\/p>\n
\u201cThat\u2019s the good news,\u201d Prince said at the panel. \u201cWith OpenAI, it\u2019s 750 times harder. With Anthropic, it\u2019s 36,000 times harder to get traffic.\u201d\u00a0<\/p>\n
So, a year ago, Cloudflare announced AI Audit, which let customers see when AI bots were scraping their site for data and let them choose whether to allow it. It was an opening move meant to let users pause the sudden and rampant practice of AI crawling. Since then, over a million internet domains opted to \u201cblock all,\u201d the most aggressive option, Cohen said.\u00a0<\/p>\n
Then, on July 1 this year, Cloudflare declared it Content Independence Day, introducing a couple more important changes.\u00a0<\/p>\n
\u201cThe first thing we did was we changed the defaults on the Cloudflare network,\u201d Cohen said. \u201cAnd so what does that mean for all new domains and new customers? We block AI training. It is in the signup flow \u2026 You can decide whether you want to leave that on, or you can turn it off.\u201d<\/p>\n
Essentially, an LLM that\u2019s in its early development phase needs information in order to build itself into a coherent and useful tool. This move gave users the option to block AIs from using their website as part of that training phase.\u00a0<\/p>\n
The second thing was to increase the granularity of control that website owners have regarding the bots on their site, giving them the ability to allow or block them depending on what they are there to do, Cohen said. For example, when an LLM answers a user\u2019s question, it uses the information it already knows and also tries to search the internet for the most up-to-date info, too. This move could allow that sort of traffic if the user wants it.\u00a0<\/p>\n
The third and most eye-catching change was the introduction of a beta program for what Cloudflare calls \u201cPay Per Crawl\u201d \u2014 essentially, a website owner can choose to let a bot onto their site to scrape their data if the bot\u2019s owner will pay their asking price to do so, done via what\u2019s called a \u201c402 request.\u201d<\/p>\n
Cohen said there\u2019s been a high demand for the pay per crawl functionality since it launched. To give a sense of the scale:<\/p>\n
\u201cEvery day, there are a billion 402 requests on the Cloudflare network,\u201d Cohen said.\u00a0<\/p>\n
Billion, with a B.\u00a0\u00a0<\/p>\n
The 402 requests can be personalized to include information for a crawler to bring back to its owner about how to contact the content creator so humans can negotiate a price. The whole thing can also be automatic, allowing users to set a price per crawl at the point of the request, or a bot may be authorized to pay for up to a certain amount. It\u2019ll fulfill the 402 request then and there in that case.\u00a0<\/p>\n
\u201cThe idea around all this stuff is that we believe that the site owners have the right to transparency, to know who is on their site,\u201d Cohen said. \u201cThey have the right to know what they\u2019re doing on the site, and then they have the right to decide whether or not they should be (there), and maybe they want to allow them for free or maybe they want to charge them.\u201d<\/p>\n
This ability for creators to have a say over who can access content, rather than a blanket assumption from AI companies that they can access anything they want, creates scarcity, Cohen said. That then creates a new kind of marketplace \u2014 one in which original work, such as local journalism, can recover some of the work\u2019s monetary value online that AI and other factors have eroded.<\/p>\n
\u201cI think local news is really interesting because it\u2019s something that people really care about right in a specific area,\u201d Cohen said. \u201cI do think that it is highly valuable to a bunch of these AI companies to have really good consumer experiences. And having really good consumer experiences means having access to accurate, high-quality, up-to-date information.\u201d\u00a0<\/p>\n
![\"\"](\"https:\/\/www.newsbeep.com\/au\/wp-content\/uploads\/2025\/09\/Fortune-Brainstorm-Tech-2025-panel-.png\")
Andrew Nusca, left, moderates a panel at the Fortune Brainstorm Tech 2025 conference titled \u201cSearch Engine Zero: Who Controls the Future of Discovery\u201d that included, from left, CEO Of People Inc. Neil Vogel, Cloudflare CEO and Park Record co-owner Matthew Prince, Editor-in-Chief and CEO of Ankler Media Janice Min and ProRata.ai founder and CEO Bill Gross. Credit: Fortune<\/p>\n
On that same Fortune Brainstorm Tech panel, Neil Vogel, CEO of People Inc., which says it is the largest digital and print publisher in America, said that being able to block AI traffic gives content creators and publishers the necessary leverage to give them a seat at the table with AI companies like OpenAI.\u00a0<\/p>\n
\u201cSome of the AI shops are good actors,\u201d Vogel said. \u201cOpenAI, we have a commercial deal with them. We\u2019re very pleased with it. We do a lot of work with them.\u201d<\/p>\n
Bill Gross was on the panel representing his AI company, ProRata.ai, which employs a revenue sharing model with content creators on a pay-per-use basis and markets itself as a solution to concerns about AI traffic. Gross likened the way AI companies crawl content to the early days of YouTube.\u00a0<\/p>\n
\u201cIf you look back at \u2026 when YouTube first started, people were stealing content left and right and posting it. There were take down notices every day from the studios,\u201d he said. \u201cAnd then finally, YouTube said, \u2018How about if we share revenue with you?\u2019 And the studio said, \u2018Great, we\u2019ll give you all this stuff.\u2019 And it opened up a whole range of explosions of great content.\u201d\u00a0<\/p>\n
Google\u2019s AI, in contrast, Vogel said, is a particularly bad actor because it uses one AI crawler for everything it does: Google\u2019s AI is the same if it\u2019s searching a site to index it in order to show it in a return of search results or if it\u2019s just seeking an answer for an AI Overview result. If you cut off the crawler at all, you lose search traffic, too. Right now, it\u2019s all or nothing with Google, even with Cloudflare\u2019s ability to give users more granular control over AI bots.\u00a0<\/p>\n
Prince, who is openly critical of Google, was bullish:\u00a0<\/p>\n
\u201cMy prediction is that by this time next year, Google will be paying content creators for crawling their content and taking it and putting in AI models,\u201d Prince said.<\/p>\n
His hope for the future of the internet is swiss cheese: as AI companies catalogue what does exist to scrape, they\u2019ll also uncover what doesn\u2019t exist \u2014 the holes in the model that reveal areas of potential interest for users.\u00a0\u00a0<\/p>\n
\u201cAnd so I think the business model of the future is actually to say to content creators, here are the holes in the cheese, and if you fill those holes, then we\u2019ll actually compensate you and pay you for that,\u201d Prince said.<\/p>\n
An internet built on compensating creators who fill those demands would be more beneficial to humanity than a purely traffic-driven model of the internet, Prince said, in which whatever grabs your attention is the most rewarded. That\u2019s the dynamic Google has encouraged and profited from, sometimes at the expense of a healthy information ecosystem, he said.\u00a0<\/p>\n
\u201cWhy do we have a rise in populism? Why do we have a more divided society?\u201d Prince said. \u201cIn part, it\u2019s because the business model of media has gone down the direction of, how do you get as much traffic as possible? There is an opportunity going forward that we can actually get back to rewarding original content creation.\u201d<\/p>\n
Another panelist, Janice Min, CEO of Ankler Media, which publishes multiple entertainment industry-focused newsletters via Substack, said in contrast to Vogel\u2019s desire to work with AI companies that she was more skeptical of striking deals.<\/p>\n
\u201cOur value is that we can create information that people will pay a lot of money for,\u201d Min said. \u201cI don\u2019t see the benefit to us, and I\u2019m willing to be dissuaded, but I don\u2019t see the benefit to us partnering with any AI company right now.\u201d<\/p>\n
Publishing is always hard, and the check from tech companies always looks good, Min said, but her experience in media over the years has told her not to negotiate with the giants.\u00a0<\/p>\n
\u201cWhat happened to publishing 10 years ago \u2026 the Metas, the Googles, they just wanted to come, made it attractive, and then they rolled the carpet back up and took everything, including, primarily, the advertising,\u201d Min said. \u201cSo I feel like I have to defend our audience. I have to defend our content.\u201d<\/p>\n
According to Cohen, the AI answer model is probably here to stay. Whether it\u2019s tenable is a matter of making creators equal players with the companies who want their content.\u00a0<\/p>\n
\u201cI think a lot of consumers prefer that customer experience (of AI answers). And so we\u2019ve just got to figure out an ecosystem and a model that works for that. Because that only works insofar as you have really great, high-quality content you have access to, and so there\u2019s got to be an incentive for that high-quality content to get produced,\u201d she said. \u201cSo we\u2019ve just got to figure out something that works.\u201d<\/p>\n
As for what the shift to pay per crawl could mean for local journalism, according to Don Rogers, editor and co-publisher of The Park Record, it could set a positive precedent on the business side, maybe leading to a healthier industry. <\/p>\n
\u201cThe big problem for local journalism isn\u2019t the journalism, but the business with the major aggregators taking our content and using it to help undercut the funding sources for it,\u201d Rogers said. \u201cThey do not reinvest in the journalism, and so here we are, with three quarters of the journalists gone and replaced by commentators, basically.\u201d<\/p>\n
And, practically speaking, forcing AI companies to work out a fair fee for use is better than trying to sue them or turning to government solutions, Rogers said, such as with The New York Times, who sued OpenAI for using the Times\u2019 content for model training.\u00a0<\/p>\n
\u201cTo me, it suggests a more elegant solution than countries like Australia and Canada trying to tax a Facebook or a Google for the content they, in essence, steal,\u201d he said.<\/p>\n
As for how this change could impact The Park Record specifically, Rogers said the paper is \u201ca drop in the proverbial ocean.\u201d\u00a0<\/p>\n
\u201cHonestly, I think we\u2019re too small to move any dials,\u201d he said. \u201cI just really like the implications that should encompass all the social media companies and content aggregators ripping off content and starving actual journalism at the root,\u201d he said.<\/p>\n
\n\tRelated Stories<\/p>\n","protected":false},"excerpt":{"rendered":"You might have noticed the shift start about May last year.\u00a0 That\u2019s when Google started implementing its AI…\n","protected":false},"author":2,"featured_media":170703,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[18],"tags":[256,64,63,237,1996,105],"class_list":{"0":"post-170702","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-internet","8":"tag-ai","9":"tag-au","10":"tag-australia","11":"tag-internet","12":"tag-tech","13":"tag-technology"},"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/170702","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/comments?post=170702"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/170702\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media\/170703"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media?parent=170702"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/categories?post=170702"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/tags?post=170702"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}