You might have noticed the shift start about May last year.
That’s when Google started implementing its AI Overview feature, which showed users an AI-powered, aggregated response to searches rather than just listing relevant websites.
“What was going on is that you’re reading the derivative rather than the original,” said Stephanie Cohen, a Parkite and the chief strategic officer for Cloudflare, an information and communications technology company. “So we were going from a world of search engines to a world of ‘answer engines.’”
Suddenly, you didn’t have to click through to a website to get the information you were searching for, even for answers to relatively complex questions. AI Overview would come up with one, courtesy of all the websites it had indexed, free of charge to Google.
And before that, ChatGPT, the popular large language model — or AI, if you want to feel like you live in the future — first launched in November 2022. The service reached 100 million users by January 2023, according to The Guardian, and like many popular AI models, it can also return synthesized answers to user queries based on trawling the internet.
Whatever you think of that, Google and AI companies have for the last few years benefited massively, treating the internet like a free buffet by scraping information off websites and condensing it into pithy answers — before anyone could say otherwise.
The end result, which is probably here to stay, is a shift in how we use the internet, likely more convenient for the user. But what’s the cost to the people who feed the machine? How are writers, artists and journalists footing the bill of this new paradigm, and can AI companies be made to pay their fair share?
Take a newspaper like The Park Record, for instance. Original reporting, while still valuable as print media, has also become fodder for the aggregate machine of the online world. That’s a change to the internet’s longtime business model, one which Google had originally helped facilitate: a search engine points to a website, and the website can sell ads or charge a subscription — or not. That was the deal.
Credit: Cloudflare
As that dynamic has changed, Cloudflare is one company seeking to rebalance content creators’ leverage in the AI era.
The California-based company works with customers to build and run applications “so they’re fast and secure,” Cohen said, claiming that more than 20% of the internet is built with Cloudflare. Small companies, big companies, free customers and paying customers, a lot of people rely on Cloudflare, including The Park Record. After all, Cloudflare’s CEO and co-founder is Matthew Prince, co-owner of The Park Record with his wife, Tatiana.
Cloudflare’s services are the kinds of things that will make sense to you if you’re computer savvy, and sound like nonsense otherwise: content delivery networks, web access filters, DDoS mitigation — stuff that’s important to the internet running smoothly.
For talking about AI, the company’s bot management services are most important, said Cohen.
“Basically, (we) make sure good traffic gets through as fast and securely as possible,” she explained, “and we deflect bad traffic, traffic you don’t want on your site. We use that same network to protect employees … your data, your network on the internet.”
Good in, bad out. Around when Google started implementing AI Overview, Cloudflare started hearing from its customers.
“The way that traffic was coming to (customers’ sites) was changing, and my guess is that’s true for The Park Record; it was true for publishers and content creators of all sizes,” Cohen said.
They dug into the data and found that what customers were saying was exactly right, Cohen said. AI was eating up traffic.
“And if you thought about it and you thought about your own personal life, what was going on is that you’re reading the derivative rather than the original,” she said. “But you still need the original source to have a thriving internet, to have an internet that helps you answer these types of questions. And so we realized that this was going to increasingly be a problem.”
More specifically, Prince, on a panel at the recent Fortune Brainstorm Tech 2025 — a tech-focused conference that Fortune Live Media on Sept. 8-10 at the Montage Deer Valley — said that the AI Overview feature has made it “10 times harder to get traffic from Google than it was just a few years ago.”
“That’s the good news,” Prince said at the panel. “With OpenAI, it’s 750 times harder. With Anthropic, it’s 36,000 times harder to get traffic.”
So, a year ago, Cloudflare announced AI Audit, which let customers see when AI bots were scraping their site for data and let them choose whether to allow it. It was an opening move meant to let users pause the sudden and rampant practice of AI crawling. Since then, over a million internet domains opted to “block all,” the most aggressive option, Cohen said.
Then, on July 1 this year, Cloudflare declared it Content Independence Day, introducing a couple more important changes.
“The first thing we did was we changed the defaults on the Cloudflare network,” Cohen said. “And so what does that mean for all new domains and new customers? We block AI training. It is in the signup flow … You can decide whether you want to leave that on, or you can turn it off.”
Essentially, an LLM that’s in its early development phase needs information in order to build itself into a coherent and useful tool. This move gave users the option to block AIs from using their website as part of that training phase.
The second thing was to increase the granularity of control that website owners have regarding the bots on their site, giving them the ability to allow or block them depending on what they are there to do, Cohen said. For example, when an LLM answers a user’s question, it uses the information it already knows and also tries to search the internet for the most up-to-date info, too. This move could allow that sort of traffic if the user wants it.
The third and most eye-catching change was the introduction of a beta program for what Cloudflare calls “Pay Per Crawl” — essentially, a website owner can choose to let a bot onto their site to scrape their data if the bot’s owner will pay their asking price to do so, done via what’s called a “402 request.”
Cohen said there’s been a high demand for the pay per crawl functionality since it launched. To give a sense of the scale:
“Every day, there are a billion 402 requests on the Cloudflare network,” Cohen said.
Billion, with a B.
The 402 requests can be personalized to include information for a crawler to bring back to its owner about how to contact the content creator so humans can negotiate a price. The whole thing can also be automatic, allowing users to set a price per crawl at the point of the request, or a bot may be authorized to pay for up to a certain amount. It’ll fulfill the 402 request then and there in that case.
“The idea around all this stuff is that we believe that the site owners have the right to transparency, to know who is on their site,” Cohen said. “They have the right to know what they’re doing on the site, and then they have the right to decide whether or not they should be (there), and maybe they want to allow them for free or maybe they want to charge them.”
This ability for creators to have a say over who can access content, rather than a blanket assumption from AI companies that they can access anything they want, creates scarcity, Cohen said. That then creates a new kind of marketplace — one in which original work, such as local journalism, can recover some of the work’s monetary value online that AI and other factors have eroded.
“I think local news is really interesting because it’s something that people really care about right in a specific area,” Cohen said. “I do think that it is highly valuable to a bunch of these AI companies to have really good consumer experiences. And having really good consumer experiences means having access to accurate, high-quality, up-to-date information.”
Andrew Nusca, left, moderates a panel at the Fortune Brainstorm Tech 2025 conference titled “Search Engine Zero: Who Controls the Future of Discovery” that included, from left, CEO Of People Inc. Neil Vogel, Cloudflare CEO and Park Record co-owner Matthew Prince, Editor-in-Chief and CEO of Ankler Media Janice Min and ProRata.ai founder and CEO Bill Gross. Credit: Fortune
On that same Fortune Brainstorm Tech panel, Neil Vogel, CEO of People Inc., which says it is the largest digital and print publisher in America, said that being able to block AI traffic gives content creators and publishers the necessary leverage to give them a seat at the table with AI companies like OpenAI.
“Some of the AI shops are good actors,” Vogel said. “OpenAI, we have a commercial deal with them. We’re very pleased with it. We do a lot of work with them.”
Bill Gross was on the panel representing his AI company, ProRata.ai, which employs a revenue sharing model with content creators on a pay-per-use basis and markets itself as a solution to concerns about AI traffic. Gross likened the way AI companies crawl content to the early days of YouTube.
“If you look back at … when YouTube first started, people were stealing content left and right and posting it. There were take down notices every day from the studios,” he said. “And then finally, YouTube said, ‘How about if we share revenue with you?’ And the studio said, ‘Great, we’ll give you all this stuff.’ And it opened up a whole range of explosions of great content.”
Google’s AI, in contrast, Vogel said, is a particularly bad actor because it uses one AI crawler for everything it does: Google’s AI is the same if it’s searching a site to index it in order to show it in a return of search results or if it’s just seeking an answer for an AI Overview result. If you cut off the crawler at all, you lose search traffic, too. Right now, it’s all or nothing with Google, even with Cloudflare’s ability to give users more granular control over AI bots.
Prince, who is openly critical of Google, was bullish:
“My prediction is that by this time next year, Google will be paying content creators for crawling their content and taking it and putting in AI models,” Prince said.
His hope for the future of the internet is swiss cheese: as AI companies catalogue what does exist to scrape, they’ll also uncover what doesn’t exist — the holes in the model that reveal areas of potential interest for users.
“And so I think the business model of the future is actually to say to content creators, here are the holes in the cheese, and if you fill those holes, then we’ll actually compensate you and pay you for that,” Prince said.
An internet built on compensating creators who fill those demands would be more beneficial to humanity than a purely traffic-driven model of the internet, Prince said, in which whatever grabs your attention is the most rewarded. That’s the dynamic Google has encouraged and profited from, sometimes at the expense of a healthy information ecosystem, he said.
“Why do we have a rise in populism? Why do we have a more divided society?” Prince said. “In part, it’s because the business model of media has gone down the direction of, how do you get as much traffic as possible? There is an opportunity going forward that we can actually get back to rewarding original content creation.”
Another panelist, Janice Min, CEO of Ankler Media, which publishes multiple entertainment industry-focused newsletters via Substack, said in contrast to Vogel’s desire to work with AI companies that she was more skeptical of striking deals.
“Our value is that we can create information that people will pay a lot of money for,” Min said. “I don’t see the benefit to us, and I’m willing to be dissuaded, but I don’t see the benefit to us partnering with any AI company right now.”
Publishing is always hard, and the check from tech companies always looks good, Min said, but her experience in media over the years has told her not to negotiate with the giants.
“What happened to publishing 10 years ago … the Metas, the Googles, they just wanted to come, made it attractive, and then they rolled the carpet back up and took everything, including, primarily, the advertising,” Min said. “So I feel like I have to defend our audience. I have to defend our content.”
According to Cohen, the AI answer model is probably here to stay. Whether it’s tenable is a matter of making creators equal players with the companies who want their content.
“I think a lot of consumers prefer that customer experience (of AI answers). And so we’ve just got to figure out an ecosystem and a model that works for that. Because that only works insofar as you have really great, high-quality content you have access to, and so there’s got to be an incentive for that high-quality content to get produced,” she said. “So we’ve just got to figure out something that works.”
As for what the shift to pay per crawl could mean for local journalism, according to Don Rogers, editor and co-publisher of The Park Record, it could set a positive precedent on the business side, maybe leading to a healthier industry.
“The big problem for local journalism isn’t the journalism, but the business with the major aggregators taking our content and using it to help undercut the funding sources for it,” Rogers said. “They do not reinvest in the journalism, and so here we are, with three quarters of the journalists gone and replaced by commentators, basically.”
And, practically speaking, forcing AI companies to work out a fair fee for use is better than trying to sue them or turning to government solutions, Rogers said, such as with The New York Times, who sued OpenAI for using the Times’ content for model training.
“To me, it suggests a more elegant solution than countries like Australia and Canada trying to tax a Facebook or a Google for the content they, in essence, steal,” he said.
As for how this change could impact The Park Record specifically, Rogers said the paper is “a drop in the proverbial ocean.”
“Honestly, I think we’re too small to move any dials,” he said. “I just really like the implications that should encompass all the social media companies and content aggregators ripping off content and starving actual journalism at the root,” he said.
Related Stories