# # # http://searchenginewatch.com/article/2067357/Bye-bye-Crawler-Blocking-the-Parasites # # #Yandex (RU) #Info: http://yandex.com/bots gives us no information on Yandex-specific robots.txt usage. User-agent: Yandex Disallow: / #Goo (JP) #Info (English): http://help.goo.ne.jp/help/article/853/ #User-agent: moget #User-agent: ichiro #Disallow: / #Naver (KR) #Info: http://help.naver.com/customer/etc/webDocument02.nhn #User-agent: NaverBot #User-agent: Yeti #Disallow: / #Baidu (CN) #Info: http://www.baidu.com/search/spider.htm #User-agent: Baiduspider #User-agent: Baiduspider-video #User-agent: Baiduspider-image #Disallow: / #SoGou (CN) #Info: http://www.sogou.com/docs/help/webmasters.htm#07 User-agent: sogou spider Disallow: / # Banned this by IP address since it did not respect the robots.txt files in # CSF on 3/27/12 Their IP banned was 220.181.94.225 #Youdao (CN) #Info: http://www.youdao.com/help/webmaster/spider/ #User-agent: YoudaoBot #Disallow: / #User-agent: moget #User-agent: ichiro #Disallow: / User-agent: bingbot Disallow: Crawl-delay: 10 User-Agent: * Allow: / Allow: /business-solutions Allow: /get-highlighter Allow: /collection Allow: /widget Allow: /plugins Allow: /blog Allow: /go Allow: /videos Allow: /tour Allow: /demo Allow: /site/ Allow: /site/www.craigslist.org/ Allow: /site/www.ebay.com/ Allow: /site/links.html Allow: /links.html Crawl-delay: 10