\n
\n
\n
![\"A](\"https:\/\/www.newsbeep.com\/us\/wp-content\/uploads\/2026\/11\/intro-1793442580.jpg\")
<\/p>\nThe new robots.txt directive will prevent the companies from scraping content for generative AI training. <\/p>\n
\n
\n
\n
<\/p>\n
tadamichi\/Shutterstock<\/p>\n
Google<\/a>, Anthropic, OpenAI<\/a> and Microsoft have committed<\/a> to honoring a new robots.txt directive, which will allow publishers to specify their preferences when it comes to the scraping of their content for\u00a0generative AI training. The companies have decided to adopt a proposal<\/a> under consideration at IETF, the\u00a0open standards organization for the internet, written by Anthropic technical lead for standards\u00a0Kevin Kelley.\u00a0<\/p>\n As OpenAI and Google explained in their announcements, website owners currently have to identify individual crawlers and add separate rules for each provider. Kelley’s proposal will change that by getting multiple AI companies to recognize and honor a single common signal.\u00a0Publishers have to set up a new user agent in the robots.txt file that’s specifically for AI training, which will look like this:<\/p>\n User-Agent: __usage_ai-training That will block crawlers specifically meant to gather data for AI training. The “Allow” section gives website operators the option to allow scraping for AI training for specific parts of the page, such as if there’s a press kit on it. But this new user-agent section doesn’t prevent crawling altogether or prevent content from showing up on Google search. For instance, Google will specifically interpret it as a stop sign\u00a0specifically for the “Google-Extended” user agent, which\u00a0crawls websites for materials used to train\u00a0Gemini models.\u00a0<\/p>\n Another thing to note is that this directive cannot undo any past scraping the companies have already done. If they’d crawled a page previously, they already have its content. This will just prevent them from scraping websites or specific pages in the future. Anthropic has already started supporting<\/a>\u00a0the new directive, while Microsoft<\/a> said it will start supporting it\u00a0on February 1, 2027. Google said it will support it in early 2027, as well, while OpenAI pledged to implement the new signal in the coming months.\u00a0<\/p>\n","protected":false},"excerpt":{"rendered":"The new robots.txt directive will prevent the companies from scraping content for generative AI training.
…\n","protected":false},"author":2,"featured_media":902058,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[45],"tags":[182,181,507,74],"class_list":["post-902057","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/posts\/902057","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/comments?post=902057"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/posts\/902057\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/media\/902058"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/media?parent=902057"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/categories?post=902057"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/tags?post=902057"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
Disallow: \/
Allow: \/press-kit\/<\/p>\n