{"id":617487,"date":"2026-09-16T04:34:09","date_gmt":"2026-09-16T04:34:09","guid":{"rendered":"https:\/\/www.newsbeep.com\/il\/617487\/"},"modified":"2026-09-16T04:34:09","modified_gmt":"2026-09-16T04:34:09","slug":"cloudflare-lets-websites-block-ai-training-without-losing-se","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/il\/617487\/","title":{"rendered":"Cloudflare Lets Websites Block AI Training Without Losing Se\u2026"},"content":{"rendered":"

Connectivity cloud company Cloudflare<\/a> has launched a setting that enables website owners to remain visible in search results while blocking crawlers from using their content for artificial intelligence training, the company said in a Tuesday (Sept. 15) press release<\/a>.<\/p>\n

Cloudflare\u2019s new Disallow AI Training setting is designed to solve a problem presented by today\u2019s mixed-use crawlers, which collect content for both a search index and AI training. Until now, website owners who blocked these crawlers from AI training also blocked them from search, according to the release.<\/p>\n

Now, if a website owner uses Disallow AI Training to refuse AI training, mixed-use crawlers will be allowed to access that site only if they honor that choice, per the release.<\/p>\n

\"Preferred<\/p>\n

We\u2019d love to be your preferred source for news<\/a>.<\/p>\n

Please add us to your preferred sources list so our news, data and interviews show up in your feed. Thanks!<\/p>\n

Apple<\/a>, Google<\/a> and Microsoft<\/a> have each either met the criteria required by the setting or have committed to do so. Several other companies operate separate crawlers for search and training, which allows Cloudflare to block their training crawlers without affecting the search ones, according to the release.<\/p>\n

\u201cThis is how we make the internet better: preserving the openness that makes search valuable while giving the people and businesses behind the web meaningful control over how their work is used,\u201d Cloudflare Co-Founder and CEO Matthew Prince<\/a> said in the release. \u201cWe look forward to continued collaborative engagement with companies like Apple, Google and Microsoft as we work together to build a healthy ecosystem.\u201d<\/p>\n

Bryan Becker<\/a>, director of product, appsec integrity and trust solutions at Cloudflare, said in a Tuesday blog post<\/a> that Cloudflare has been talking with operators of crawlers about this issue since July.<\/p>\n

\u201cThe response has been encouraging: Almost all agreed that site owners should have control and transparency into how their content is used, and reassurance that their choices will be respected,\u201d Becker said.<\/p>\n

The PYMNTS Intelligence<\/a> report \u201cHow Enterprises Can Build a \u2018Know Your Agent\u2019 Defense: Digital Identity Verification in the Age of Bots<\/a>\u201c found that nearly 90% of organizations said managing bot activity is a challenge.<\/p>\n

As AI-powered agents interact with platforms alongside human users, weak or outdated controls allow malicious bots to slip through while legitimate users are mistakenly blocked, the report found.<\/p>\n","protected":false},"excerpt":{"rendered":"Connectivity cloud company Cloudflare has launched a setting that enables website owners to remain visible in search results…\n","protected":false},"author":2,"featured_media":617488,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[345,12918,114,8022,85,46,43,262,264],"class_list":["post-617487","post","type-post","status-publish","format-standard","has-post-thumbnail","category-business","tag-ai","tag-ai-training","tag-business","tag-cloudflare","tag-il","tag-israel","tag-news","tag-pymnts-news","tag-whats-hot"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/posts\/617487","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/comments?post=617487"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/posts\/617487\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/media\/617488"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/media?parent=617487"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/categories?post=617487"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/tags?post=617487"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}