Robots.txt

A text file stored in the top level directory of a web site to deny access by robots to certain pages or sub-directories of the site. Only robots which comply with the Robots Exclusion Standard will read and obey the commands in this file. Robots will read this file on each visit, so that pages or areas of sites can be made public or private at any time by changing the content of robots.txt before re-submitting to the search engines. The simple example below attempts to prevent all robots from visiting the /secret directory:

User-agent: *

Disallow: /secret

A file which sits in the root of a site and tells search engines which files not to crawl. Some search engines will still list your URLs as URL only listings even if you block them using a robots.txt file.

Do not put files on a public server if you do not want search engines to index them!

See also:

• Robotstxt.org

Text file placed in a websites root directory and linked in the html code. Allows for SEO's to control the actions of search engine spiders on the site or even deny them access.