![\"Stop](\"https:\/\/www.newsbeep.com\/ca\/wp-content\/uploads\/2025\/07\/1753270277_558_960x0.jpg\")
<\/p>\n<\/p>\n
Stop AI from scraping data for free<\/p>\n
getty<\/p>\n
As someone who has spent years building partnerships between tech innovators and digital creators, I\u2019ve seen how difficult it can be to balance visibility and value. Every week, I meet with founders and business leaders trying to figure out how to stand out, monetize content, and keep control of their digital assets. They\u2019re proud of what they\u2019ve built but increasingly worried that AI systems are consuming their work without permission, credit, or compensation.<\/p>\n
That\u2019s why Cloudflare\u2019s latest announcement hit like a thunderclap. And I wanted to wait to see the responses from companies and creators to really tell this story.<\/p>\n
Cloudflare, one of the internet\u2019s most important infrastructure companies, now blocks AI crawlers by default for all new customers.<\/p>\n
This flips the longstanding model, where crawlers were allowed unless actively blocked, into something more deliberate: AI must now ask to enter.<\/p>\n
And not just ask. Pay.<\/p>\n
Cloudflare wants AI Companies to pay for the data that they use. <\/p>\n
getty<\/p>\n
Alongside that change, Cloudflare has launched Pay\u2011Per\u2011Crawl, a new marketplace that allows website owners to charge AI companies per page crawled. If you\u2019re running a blog, a digital magazine, a startup product page, or even a knowledge base, you now have the option to set a price for access. AI bots must identify themselves, send payment, and only then can they index your content.<\/p>\n
This isn\u2019t a routine product update. It\u2019s a signal that the free ride for AI training data is ending and a new economic framework is beginning.<\/p>\n
AI Models and Their Training<\/p>\n
The core issue behind this shift is how AI models are trained. Large language models like OpenAI\u2019s GPT or Anthropic\u2019s Claude rely on huge amounts of data from the open web. They scrape everything, including articles, FAQs, social posts, documentation, even Reddit threads, to get smarter. But while they benefit, the content creators see none of that upside.<\/p>\n
Unlike traditional search engines that drive traffic back to the sites they crawl, generative AI tends to provide full answers directly to users, cutting creators out of the loop.<\/p>\n
According to Cloudflare, the data is telling: <\/a>OpenAI\u2019s crawl-to-referral ratio is around 1,700 to 1. Anthropic\u2019s is 73,000 to 1. Compare that to Google, which averages about 14 crawls per referral, and the imbalance becomes clear.<\/p>\n In other words, AI isn\u2019t just learning from your content but it\u2019s monetizing it without ever sending users back your way.<\/p>\n Rebalancing the AI Equation<\/p>\n Cloudflare\u2019s announcement aims to rebalance this equation. From now on, when someone signs up for a new website using Cloudflare\u2019s services, AI crawlers are automatically blocked unless explicitly permitted. For existing customers, this is available as an opt-in.<\/p>\n More importantly, Cloudflare now enables site owners to monetize their data through Pay\u2011Per\u2011Crawl. AI bots must:<\/p>\n Cryptographically identify themselves Only then will the content be served.<\/p>\n We have to rebalance the AI equation.<\/p>\n getty<\/p>\n This marks a turning point. Instead of AI companies silently harvesting the web, they must now enter into economic relationships with content owners. The model is structured like a digital toll road and this road leads to your ideas, your writing, and your value.<\/p>\n Several major publishers are already onboard. According to Neiman Lab<\/a>, Gannett, Cond\u00e9 Nast, The Atlantic, BuzzFeed, Time, and others have joined the system to protect and monetize their work.<\/p>\n Cloudflare Isn\u2019t The Only One Trying To Protect Creators From AI<\/p>\n This isn\u2019t happening in a vacuum. A broader wave of startups and platforms are emerging to support a consent-based data ecosystem.<\/p>\n CrowdGenAI is focused on assembling ethically sourced, human-labeled data that AI developers can license with confidence. It\u2019s designed for the next generation of AI training where the value of quality and consent outweighs quantity. (Note: I am on the advisory board of CrowdGenAI).<\/p>\n Real.Photos is a mobile camera app that verifies your photos are real, not AI. The app also verifies where the photo was taken and when. The photo, along with its metadata are hashed so it can’t be altered. Each photo is stored on the Base blockchain as an NFT and the photo can be looked up and viewed on a global, public database. Photographers make money by selling rights to their photos. (Note: the founder of Real.Photos is on the board of Unstoppable – my employer)<\/p>\n Spawning.ai gives artists and creators control over their inclusion in datasets. Their tools let you mark your work as \u201cdo not train,\u201d with the goal of building a system where creators decide whether or not they\u2019re part of AI\u2019s learning process.<\/p>\n Tonic.ai helps companies generate synthetic data for safe, customizable model training, bypassing the need to scrape the web altogether.<\/p>\n DataDistil is building a monetized, traceable content layer where AI agents can pay for premium insights, with full provenance and accountability.<\/p>\n Each of these players is pushing the same idea: your data has value, and you deserve a choice in how it\u2019s used.<\/p>\n What Are the Pros to Cloudflare\u2019s AI Approach?<\/p>\n There are real benefits to Cloudflare\u2019s new system.<\/p>\n First, it gives control back to creators. The default is \u201cno,\u201d and that alone changes the power dynamic. You no longer have to know how to write a robots.txt file or hunt for obscure bot names.<\/p>\n Cloudflare handles it.<\/p>\n Second, it introduces a long-awaited monetization channel. Instead of watching your content get scraped for free, you can now set terms and prices.<\/p>\n Third, it promotes transparency. Site owners can see who\u2019s crawling, how often, and for what purpose. This turns a shadowy process into a visible, accountable one.<\/p>\n Finally, it incentivizes AI developers to treat data respectfully. If access costs money, AI systems may start prioritizing quality, licensing, and consent.<\/p>\n And There Are Some Limitations To The AI Approach<\/p>\n But there are limitations.<\/p>\n Today, all content is priced equally. That means a one-sentence landing page costs the same to crawl as an investigative feature or technical white paper. A more sophisticated pricing model will be needed to reflect actual value.<\/p>\n Enforcement could also be tricky.<\/p>\n
\n Indicate which pages they want to access
\n Accept a price per page
\n Complete payment via Cloudflare<\/p>\n