Google-Extended, the new standalone product token, to tell Google through your robots.txt to not use your site content for Bard and Vertex AI and other AI projects does not work for the AI-answers and snapshots provided in the Search Generative Experience.
Google told us that Google-Extended does not work for SGE. “SGE is a Search experiment so website administrators should continue to use the Googlebot user agent through robots.txt and the NOINDEX meta tag to manage their content in search results, including experiments like Search Generative Experience,” a Google spokesperson told us.
Why not for SGE. Google explained that SGE is part of the Google Search experience; it is a search feature and thus it should work as how normal search directives work. “The context is that AI is built into Search, not bolted on, and integral to how Search functions, which is why robots.txt is the control to give web publishers the option to manage access to how their sites are crawled,” Google told us.
SGE AI-answers. There are examples of SGE showing AI-generated answers, with website cards, from sites that have specifically directed Google not to use their content for AI purposes.
Glenn Gabe posted a screenshot of VentureBeat.com that specifically has Google-Extended disallowed in its robot.txt file, being used in the AI-generated answer in SGE:
Why we care. If you thought using Google-Extended would prevent Google from using your content for SGE AI-powered answers, then you were wrong. Google-Extended it for Bard and Vertex AI and other non-search-specific products.
If you don’t want your answers in SGE, then you would need to block Googlebot completely, which most of you do not want to do.
