Today being July 4th, I normally would avoid writing stories here but there are a lot of rumors out there that Reddit is blocking Google Search from crawling and indexing its content. It is not true – Reddit is not blocking Google Search.
I mean, at first glance, it does look like Reddit’s robots.txt file is blocking all search engines, including Google Search:
But that is just what you and I see.
If you access it from the Google Rich Results test, as if you are really accessing it from Google’s IP ranges, you see Google is indeed allowed:
Plus, just try to see if that tool can access the Reddit home page – it can and is allowed:
I mean, to think that Reddit is blocking Google is kind of wild. For Reddit to actually do that is even more insane. Reddit is getting paid by Google for that content and Reddit gets an insane amount of traffic from Google Search.
Nice investigation work Pedro Dias on X and Ryan Siddle over here.
This is what they’re showing to Google:
# Our robots.txt is for search engines
# 80legs
User-agent: 008
Disallow: /# 80legs’ new crawler
User-agent: voltron
Disallow: /User-Agent: bender
Disallow: /my_shiny_metal_assUser-Agent: Gort
Disallow: /earthUser-agent: MJ12bot…
— Pedro Dias (@pedrodias) July 4, 2024
So you can all relax, Reddit will still rank above you for all your content in Google Search…
Forum discussion at X.
