# # This is an empty robots.txt file that can be used to indicate to # WWW robots which parts of a web site should not be indexed. # # Following page has details on how to setup a robots.txt file and # related security issues: # # http://www.chami.com/tips/internet/010198I.html # # 1. Uncomment following two lines by removing "#" at the beginning to # indicate that all robots should stay away from the /cgi-bin/ folder User-agent: * User-agent: * Disallow:/cgi-bin Disallow: /cgi-bin/ Disallow: /etc/ Disallow: /logs/ Disallow: /public_ftp/ Disallow: /_private/ Disallow: /_vti_bin/ Disallow: /_vti_cnf/ Disallow: /entrophybanner/ Disallow: /images/ Disallow: /public_html/private Disallow: /public_html/_vti_bin Disallow: /public_html/_vti_cnf Disallow: /public_html/_vti-log Disallow: /public_html/_vti_pvt Disallow: /public_html/_vti_txt Disallow: /public_html/cgi_bin Disallow: /public_html/csstest/cgi-bin Disallow: /public_html/csstest/graphics Disallow: /public_html/csstest/images Disallow: /public_html/csstest/includes Disallow: /public_html/csstest/thumbs Disallow: /public_html/csstest2/cgi_bin Disallow: /public_html/csstest2/images Disallow: /public_html/csstest2/includes Disallow: /public_entropybanner/ Disallow: /public_html/fmsforum/admin/ Disallow: /public_html/fmsforum/cache/ Disallow: /public_html/fmsforum/cgi-gin/ Disallow: /public_html/fmsforum/db/ Disallow: /public_html/fmsforum/docs/ Disallow: /public_html/fmsforum/images/ Disallow: /public_html/fmsforum/includes/ Disallow: /public_html/fmsforum/schemas/ Disallow: /public_html/fmsforum/ Disallow: /public_html/fmsforum/language/ Disallow: /public_html/fmsforum/templates/ Disallow: /public_html/fmsforum/common.php Disallow: /public_html/fmsforum/config.php Disallow: /public_html/fmsforum/faq.php Disallow: /public_html/fmsforum/groupcp.php Disallow: /public_html/fmsforum/login.php Disallow: /public_html/fmsforum/membelist.php Disallow: /public_html/fmsforum/modcp.php Disallow: /public_html/fmsforum/posting.php Disallow: /public_html/fmsforum/privmsg.php Disallow: /public_html/fmsforum/profile.php Disallow: /public_html/fmsforum/profile.php Disallow: /public_html/fmsforum/search.php Disallow: /public_html/fmsforum/viewonline.php # 2. Following two lines indicate that robots named "WWW_Robot_Name" # should stay away from this site. User-Agent: findlinks/1.01 (+http://wortschatz.uni-leipzig.de/findlinks/) Disallow: / User-agent: Wget Disallow: / User-agent: wget/1.6 disallow: / User-Agent: Googlebot-Image/1.0 disallow: /images User-Agent: msnbot Crawl-Delay: 50 User-Agent: Slurp Crawl-Delay: 50 User-Agent: yahoo Crawl-Delay: 50 User-Agent: Inkotomi Crawl-Delay: 50 User-Agent: msnbot/1.0 (+http://search.msn.com/msnbot.htm) Crawl-Delay: 75 # 3. Following two lines indicate that robots named "WWW_Wanderer_Name" # should not index files in the /private/ folder. User-agent: * Disallow: /private/