User-Agent: ArchitextSpider # Excite User-Agent: Ask Jeeves User-Agent: FAST-WebCrawler # Alltheweb User-Agent: Freecrawl # euroseek.net User-Agent: Googlebot User-Agent: Mediapartners-Google* User-Agent: Gulliver # Northern Light User-Agent: ia_archiver # web.archive.org User-Agent: InfoSeek User-Agent: Lycos User-Agent: MSNBot # MSN User-Agent: Scooter # AltaVista User-Agent: Slurp # Inktomi and Yahoo Disallow: /scripts Disallow: /jobs Disallow: /admin Disallow: /test Disallow: /testing Disallow: /cron Disallow: /includes Disallow: /hearstmags Disallow: /reading Disallow: /wingsnob User-Agent: * Disallow: /