Gary Illyes from Google announced at the Google Search Central Live in Singapore yesterday that Googlebot, Google, can now crawl from outside of the US. He said, according to reports, that the system to handle this is currently live but may not be used in all cases or it may not be used at all for your site.
Update, this report is not 100% exactly accurate. Gary posted on Mastodon after he saw this story saying “that’s not quite what I said. I said that the IP blocks are allocated and you can check the countries they are assigned to, and we may use them in the future for local crawls.”
no he didn’t
— Gary 鯨理／경리 Illyes (@methode) November 25, 2022
Here is the rest of the original story as I previously reported it. But please take into account the update from Gary above.
Kenichi Suzuki, who was at the event, posted on Twitter saying “Googlebot can crawl from a country where the server is located (i.e. from outside US). The system has already gone live but may not be used (at all). In addition, it becomes effective only in a condition where access from outside of the country is blocked.”
Here is that tweet:
Googlebot can crawl from a country where the server is located (i.e. from outside US). The system has already gone live but may not be used (at all). In addition, it becomes effective only in a condition where access from outside of the country is blocked. #SearchCentralLive https://t.co/1hPl67NPw8
— Kenichi Suzuki🇺🇦鈴木謙一 (@suzukik) November 24, 2022
We heard rumors that this might happen earlier on this year from Google’s John Mueller. John said then “I could see this [where Google crawls from] expanding over time if we see a broader need.” So it looks like we are there.
Keep in mind, this is probably used for sites that (a) block Googlebot access from US regions, (b) show different content from different countries and (c) Google suspects something fishy based on geo-targeting.
Google previously said that it did not find any benefit to crawling from outside of the US back in 2017. There are however times Google will crawl from a specific region but it is rare.
Forum discussion at Twitter.
