# /robots.txt file for http://webcrawler.com/ # mail webmaster@webcrawler.com for constructive criticism User-agent: * Disallow: /cgi-bin/ Disallow: /ace/ Disallow: /acebook/ Disallow: /diamond_roofing/ Disallow: /do_boys/ Disallow: /flashtest/ Disallow: /ftp/ Disallow: /images/ Disallow: /lcd/ Disallow: /lookbooks/ Disallow: /donoho/ Disallow: /newellco/ Disallow: /NPC/ Disallow: /photos/ Disallow: /pogo/ Disallow: /st_video/ Disallow: /store/ Disallow: /suspended.page/ Disallow: /blog/wp-content/ Disallow: /blog/trackback/ Disallow: /blog/wp-admin/ Disallow: /blog/feed/ Disallow: /blog/archives/ Disallow: /blog/index.php Disallow: /blog/*? Disallow: /blog/*.php$ Disallow: /blog/*.js$ Disallow: /blog/*.inc$ Disallow: /blog/*.css$ Disallow: /blog/*/feed/ Disallow: /blog/*/trackback/ Disallow: /blog/page/ Disallow: /blog/tag/ Disallow: /blog/category/ User-agent: Googlebot-Image Disallow: /blog/wp-includes/ User-agent: Mediapartners-Google* Disallow: User-agent: ia_archiver Disallow: / User-agent: duggmirror Disallow: / Sitemap: http://www.hybr-id.com/blog/sitemap.xml