The Magic SEO Ball

All your SEO questions answered by The Magic SEO Ball!

Is Google wrong about robots.txt?

March 10, 2014 By Natan Gesher

I saw a search result that said, “A description for this result is not available because of this site’s robots.txt – learn more.” But when I checked that site’s robots.txt, the page was not blocked at all. Did Google just mess this one up?

Magic SEO Ball says: most likely.

Most likely.

I searched for Ray Kurzweil so I could explain to some coworkers that he seriously intends not to die. The fifth and sixth organic results were both from ted.com. The fifth result, but not the sixth, strongly implied that Googlebot had been blocked from crawling the page, and consequently was unable to provide a meta description.

A description for this result is not available because of this site's robots.txt – learn more.

But when I looked at ted.com’s robots.txt, I saw only this:

User-agent: *
Disallow: /index.php/profiles/browse
Disallow: /index.php/search
Disallow: /search

Google wasn’t blocked from crawling the page at all.

I don’t know what’s going on here – specifically whether TED has used another method to prevent this page from being crawled or whether Google actually thinks that the ted.com robots.txt is blocking that directory – but it seems strangely like a Google mistake.

Filed Under: Yes

All your SEO questions answered by a black magic ball!

  • About the Magic SEO Ball
  • Ask A Question
  • SEO glossary

Recent Posts

  • Are fragment identifiers that change content cloaking?
  • Does UTM tracking on inbound links affect SEO?
  • Did Thumbtack really break Google’s rules?
  • Is HTTPS a tie-breaker?
  • Is there a duplicate content penalty?

Natan Gesher | Sharav | Colossal Reviews | Megalomania:me | The Magic SEO Ball | LinkedIn | Facebook | Mastodon