If You Thought Your Robot was Uptight

31 Mar 2010 • SEO News

If You Thought Your Robot was Uptight

You know when you are searching for those golden links across sites on THAT page that you just know will stand the test of time and provide all the link juice you were looking for only to find it is a nofollow link, that`s not so bad i hear you cry, ok how about a noindex meta tag or if that is not immediately apparent how about an exclusion via Robots.txt?

So you continue to search bruised and battered on to the next potential and it leaves you a little sore knowing you have just wasted no amount of time looking for that one that is going to meet all of the criteria you need.

There are some crazy Robots.txt files out there take a look at Googles (UK) for example;

User-agent: * Disallow: /search Disallow: /groups Disallow: /images Disallow: /catalogs Disallow: /catalogues Disallow: /news Allow: /news/directory Disallow: /nwshp Disallow: /setnewsprefs? Disallow: /index.html? Disallow: /? Disallow: /addurl/image? Disallow: /pagead/ Disallow: /relpage/ Disallow: /relcontent Disallow: /imgres Disallow: /imglanding Disallow: /keyword/ Disallow: /u/ Disallow: /univ/ Disallow: /cobrand Disallow: /custom Disallow: /advanced_group_search Disallow: /googlesite Disallow: /preferences Disallow: /setprefs Disallow: /swr Disallow: /url Disallow: /default Disallow: /m? Disallow: /m/? Disallow: /m/blogs? Disallow: /m/ig Disallow: /m/images? Disallow: /m/local? Disallow: /m/movies? Disallow: /m/news? Disallow: /m/news/i? Disallow: /m/place? Disallow: /m/setnewsprefs? Disallow: /m/search? Disallow: /m/swmloptin? Disallow: /m/trends Disallow: /wml? Disallow: /wml/? Disallow: /wml/search? Disallow: /xhtml? Disallow: /xhtml/? Disallow: /xhtml/search? Disallow: /xml? Disallow: /imode? Disallow: /imode/? Disallow: /imode/search? Disallow: /jsky? Disallow: /jsky/? Disallow: /jsky/search? Disallow: /pda? Disallow: /pda/? Disallow: /pda/search? Disallow: /sprint_xhtml Disallow: /sprint_wml Disallow: /pqa Disallow: /palm Disallow: /gwt/ Disallow: /purchases Disallow: /hws Disallow: /bsd? Disallow: /linux? Disallow: /mac? Disallow: /microsoft? Disallow: /unclesam? Disallow: /answers/search?q= Disallow: /local? Disallow: /local_url Disallow: /froogle? Disallow: /products? Disallow: /products/ Disallow: /froogle_ Disallow: /product_ Disallow: /products_ Disallow: /print Disallow: /books Disallow: /bkshp?q= Allow: /booksrightsholders Disallow: /patents? Disallow: /patents/ Allow: /patents/about Disallow: /scholar Disallow: /complete Disallow: /sponsoredlinks Disallow: /videosearch? Disallow: /videopreview? Disallow: /videoprograminfo? Disallow: /maps? Disallow: /mapstt? Disallow: /mapslt? Disallow: /maps/stk/ Disallow: /maps/br? Disallow: /mapabcpoi? Disallow: /maphp? Disallow: /places/ Disallow: /maps/place Disallow: /help/maps/streetview/partners/welcome/ Disallow: /lochp? Disallow: /center Disallow: /ie? Disallow: /sms/demo? Disallow: /katrina? Disallow: /blogsearch? Disallow: /blogsearch/ Disallow: /blogsearch_feeds Disallow: /advanced_blog_search Disallow: /reader/ Allow: /reader/play Disallow: /uds/ Disallow: /chart? Disallow: /transit? Disallow: /mbd? Disallow: /extern_js/ Disallow: /calendar/feeds/ Disallow: /calendar/ical/ Disallow: /cl2/feeds/ Disallow: /cl2/ical/ Disallow: /coop/directory Disallow: /coop/manage Disallow: /trends? Disallow: /trends/music? Disallow: /notebook/search? Disallow: /musica Disallow: /musicad Disallow: /musicas Disallow: /musicl Disallow: /musics Disallow: /musicsearch Disallow: /musicsp Disallow: /musiclp Disallow: /browsersync Disallow: /call Disallow: /archivesearch? Disallow: /archivesearch/url Disallow: /archivesearch/advanced_search Disallow: /base/search? Disallow: /base/reportbadoffer Disallow: /base/s2 Disallow: /urchin_test/ Disallow: /movies? Disallow: /codesearch? Disallow: /codesearch/feeds/search? Disallow: /wapsearch? Disallow: /safebrowsing Allow: /safebrowsing/diagnostic Allow: /safebrowsing/report_error/ Allow: /safebrowsing/report_phish/ Disallow: /reviews/search? Disallow: /orkut/albums Disallow: /jsapi Disallow: /views? Disallow: /c/ Disallow: /cbk Disallow: /recharge/dashboard/car Disallow: /recharge/dashboard/static/ Disallow: /translate_a/ Disallow: /translate_c Disallow: /translate_f Disallow: /translate_static/ Disallow: /translate_suggestion Disallow: /profiles/me Allow: /profiles Disallow: /s2/profiles/me Allow: /s2/profiles Allow: /s2/photos Allow: /s2/static Disallow: /s2 Disallow: /transconsole/portal/ Disallow: /gcc/ Disallow: /aclk Disallow: /cse? Disallow: /cse/panel Disallow: /cse/manage Disallow: /tbproxy/ Disallow: /comparisonads/ Disallow: /imesync/ Disallow: /shenghuo/search? Disallow: /support/forum/search? Disallow: /reviews/polls/ Disallow: /hosted/images/ Disallow: /hosted/life/ Disallow: /ppob/? Disallow: /ppob? Disallow: /ig/add? Disallow: /adwordsresellers Disallow: /accounts/o8 Allow: /accounts/o8/id Disallow: /topicsearch?q= Disallow: /xfx7/ Disallow: /squared/api Disallow: /squared/search Disallow: /squared/table Disallow: /toolkit/ Allow: /toolkit/*.html Disallow: /qnasearch? Disallow: /errors/ Disallow: /app/updates Disallow: /sidewiki/entry/ Disallow: /quality_form? Disallow: /labs/popgadget/search Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml Sitemap: http://www.google.com/hostednews/sitemap_index.xml Sitemap: http://www.google.com/ventures/sitemap_ventures.xml Sitemap: http://www.google.com/sitemaps_webmasters.xml Sitemap: http://www.gstatic.com/trends/websites/sitemaps/sitemapindex.xml Sitemap: http://www.gstatic.com/dictionary/static/sitemaps/sitemap_index.xml

Yes i know and me and you were excluding a few image directories and maybe a cgi-bin or WP control pages. Apologies for posting the whole file but i wanted to demonstrate the lengths of control even Google is going to, obviously they have an abundance of content, more than you or me but it`s arguably the first thing the bots are looking for when it comes to content discovery, so why not take control of it?.

I am by no means an expert on Robots.txt by any shape, i get the basics and get it to do what i want it to do on a basic level, but after spending a bit of time reading through Sebastians site you get to realize the importance of taking control of how you are letting bots into your content to either index for good or basically steal your stuff. A quick look at Sebastians Robots.txt file made me laugh, Allow: /porn/ Dissallow: /smut/ I know he is testing that stuff as he clearly states but his sense of humour just makes me smile.