User-agent: Sosospider Disallow: / User-agent: 008 Disallow: / User-agent: Baiduspider Disallow: / User-agent: Yandex Disallow: / User-agent: sogou spider Disallow: / User-agent: YoudaoBot Disallow: / User-agent: NaverBot Disallow: / User-agent: Yeti Disallow: / User-agent: moget Disallow: / User-agent: ichiro Disallow: / User-agent: netseer Disallow: / User-agent: MJ12bot Disallow: / User-agent: JikeSpider Disallow: / User-agent: sistrix Disallow: / User-agent: MaxPointCrawler/Nutch-1.1 Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: NerdByNature Disallow: / User-agent: AhrefsBot Disallow: / User-agent: Acoon Disallow: / User-agent: Ezooms Disallow: / User-agent: GermCrawler Disallow: / User-agent: Exabot Disallow: / User-agent: Spinn3r Disallow: / User-agent: MLBot Disallow: / User-agent: yacybot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: magpie-crawler Disallow: / User-agent: WBSearchBot Disallow: / User-agent: YioopBot Disallow: / User-agent: discobot Disallow: / User-agent: grapeshot Disallow: / User-agent: proximic Disallow: / User-agent: dotbot Disallow: / User-agent: foobar Disallow: / User-agent: voltron Disallow: / User-agent: 008 Disallow: / User-agent: 200PleaseBot Disallow: / User-agent: 360Spider Disallow: / User-agent: Aboundexbot Disallow: / User-agent: acapbot Disallow: / User-agent: ADmantX Disallow: / User-agent: AdnormCrawler Disallow: / User-agent: AhrefsBot Disallow: / User-agent: AOLbot/4.0 Disallow: / User-agent: Attribot Disallow: / User-agent: Baiduspider-image Disallow: / User-agent: Bot.AraTurka.com Disallow: / User-agent: Butterfly/1.0 Disallow: / User-agent: BuzzSumo Disallow: / User-agent: cis455crawler Disallow: / User-agent: ClarityBot Disallow: / User-agent: ClarityDailyBot Disallow: / User-agent: Clickagy Intelligence Bot Disallow: / User-agent: CMS Crawler Disallow: / User-agent: CommonCrawler Disallow: / User-agent: crawler4j (http://code.google.com/p/crawler4j/) Disallow: / User-agent: CrazyWebCrawler Disallow: / User-agent: CrazyWebCrawler-Spider Disallow: / User-agent: DataparkSearch Disallow: / User-agent: deepcrawl Disallow: / User-agent: DeuSu Disallow: / User-agent: dialogsearch.com Disallow: / User-agent: diffbot Disallow: / User-agent: dj-research Disallow: / User-agent: DomainAppender Disallow: / User-agent: DomainMacroCrawler Disallow: / User-agent: DomainSigmaCrawler Disallow: / User-agent: Domain Re-Animator Bot Disallow: / User-agent: Ezooms Robot Disallow: / User-agent: Exabot Disallow: / User-agent: Experibot Disallow: / User-agent: FavOrg Disallow: / User-agent: Findxbot Disallow: / User-agent: FlipboardProxy Disallow: / User-agent: Friendica Disallow: / User-agent: GetProxi.es-bot Disallow: / User-agent: GigablastOpenSource/1.0 Disallow: / User-agent: Gluten Free Crawler Disallow: / User-agent: Google-HTTP-Java-Client Disallow: / User-agent: Gocrawl Disallow: / User-agent: grapeshot Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: GuzzleHttp Disallow: / User-agent: Hatena Star Disallow: / User-agent: Heritrix Disallow: / User-agent: heritrix Disallow: / User-agent: HTTrack 3 Disallow: / User-agent: ia_archiver Disallow: / User-agent: InAGist URL Resolver Disallow: / User-agent: Insitesbot Disallow: / User-agent: iskanie Disallow: / User-agent: jack Disallow: / User-agent: James BOT Disallow: / User-agent: Java Disallow: / User-agent: JS-Kit URL Resolver Disallow: / User-agent: KomodiaBot Disallow: / User-agent: linkdexbot Disallow: / User-agent: LivelapBot Disallow: / User-agent: LS Session Disallow: / User-agent: LSSRocketCrawler Disallow: / User-agent: ltx71 Disallow: / User-agent: madaali.de Disallow: / User-agent: meanpathbot Disallow: / User-agent: MegaIndex.ru Disallow: / User-agent: memoryBot Disallow: / User-agent: MetaURI API/2.0 +metauri.com Disallow: / User-agent: MJ12bot Disallow: / User-agent: Mozilla/4.0 (CMS Crawler: http://www.cmscrawler.com) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 4.0 Robot) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 5.0 Robot) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 5.0 Robot) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0 ; ClarityDailyBot) Disallow: / User-agent: Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot) Disallow: / User-agent: Mozilla/5.0 (compatible;acapbot/0.1;treat like Googlebot) Disallow: / User-agent: Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/) Disallow: / User-agent: Mozilla/5.0 (compatible; AOLbot/4.0; +http://www.aol-soft.com/) Disallow: / User-agent: Mozilla/5.0 (compatible; DomainAppender /1.0; +http://www.profound.net/domainappender) Disallow: / User-agent: Mozilla/5.0 (compatible; DomainSigmaCrawler/0.1; +http://domainsigma.com/robot) Disallow: / User-agent: Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot) Disallow: / User-agent: Mozilla/5.0 (compatible; Findxbot/1.0; +http://www.findxbot.com) Disallow: / User-agent: Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html) Disallow: / User-agent: Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/) Disallow: / User-agent: Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html) Disallow: / User-agent: Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+) Disallow: / User-agent: Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html) Disallow: / User-agent: Mozilla/5.0 (compatible; PAD-bot/9.0; +http://www.descargarprogramagratis.com/) Disallow: / User-agent: Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li) Disallow: / User-agent: Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html) Disallow: / User-agent: Mozilla/5.0 (compatible; SoftListBot/2.2; +http://www.softlist.us/) Disallow: / User-agent: Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; +http://tweetedtimes.com) Disallow: / User-agent: Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/) Disallow: / User-agent: Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: crawling@ubermetrics-technologies.com) Disallow: / User-agent: Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/) Disallow: / User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy) Disallow: / User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 - James BOT - WebCrawler http://cognitiveseo.com/bot.html Disallow: / User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com) Disallow: / User-agent: Mozilla/5.0 (Windows NT 6.2) Insitesbot/1.0 Disallow: / User-agent: MS Search 4.0 Robot Disallow: / User-agent: MS Search 5.0 Robot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: NativeHost Disallow: / User-agent: NerdyBot Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: NetShelter ContentScan Disallow: / User-agent: newsme/1.0; feedback@news.me Disallow: / User-agent: Niki-Bot Disallow: / User-agent: NING/1.0 Disallow: / User-agent: Node/simplecrawler 0.3.9 (http://github.com/cgiffard/node-simplecrawler.git) Disallow: / User-agent: OpenHoseBot Disallow: / User-agent: PAD-bot Disallow: / User-agent: panscient.com Disallow: / User-agent: PaperLiBot Disallow: / User-agent: peerindex Disallow: / User-agent: Perl LWP Disallow: / User-agent: Photon Disallow: / User-agent: plukkie Disallow: / User-agent: postano Disallow: / User-agent: proximic Disallow: / User-agent: PulseCrawler/1.1 Disallow: / User-agent: Pulsepoint XT3 web scraper Disallow: / User-agent: QuerySeekerSpider Disallow: / User-agent: quipu Disallow: / User-agent: Raven Disallow: / User-agent: RavenCrawler Disallow: / User-agent: RBot Disallow: / User-agent: Recorded Future Disallow: / User-agent: Riddler Disallow: / User-agent: Ruby Disallow: / User-agent: RukiCrawler Disallow: / User-agent: SafeDNSBot Disallow: / User-agent: Scrapy Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-agent: Searchie Disallow: / User-agent: semantic-visions.com crawler Disallow: / User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: SEOlyticsCrawler Disallow: / User-agent: ShowyouBot (http://showyou.com/crawler) Disallow: / User-agent: simplecrawler Disallow: / User-agent: SISTRIX Crawler Disallow: / User-Agent: siteluxbot Disallow: / User-agent: SmeshBot Disallow: / User-agent: SMTBot Disallow: / User-agent: srmse/Nutch Disallow: / User-agent: SoftListBot Disallow: / User-agent: sogou spider Disallow: / User-agent: spbot Disallow: / User-agent: squirrly Disallow: / User-agent: ssearch_bot Disallow: / User-agent: Stratagems Kumo Disallow: / User-agent: tbot-nutch Disallow: / User-agent: TelegramBot Disallow: / User-agent: TweetmemeBot Disallow: / User-agent: TWMBot Disallow: / User-agent: Typhoeus - https://github.com/typhoeus/typhoeus Disallow: / User-agent: uMBot-LN Disallow: / User-agent: WBSearchBot Disallow: / User-agent: WebIndex Disallow: / User-agent: VeBot Disallow: / User-agent: VeriCiteCrawler Disallow: / User-agent: WeSEE_Bot Disallow: / User-agent: WiseGuys Robot Disallow: / User-agent: voltron Disallow: / User-agent: woobot Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: X-CAD-SE Disallow: / User-agent: XoviBot Disallow: / User-agent: yacybot Disallow: / User-agent: Yahoo Pipes 1.0 Disallow: / User-agent: YioopBot Disallow: / User-agent: YadirectBot Disallow: User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /app_code/ Disallow: /bin/