The Robots Exclusion Protocol is a mechanism by which webmasters can provide information to Internet robots that visit their site. Internet Robots visiting a site can (and should) read the robots.txt file from the root directory to determine what pages they are permitted to view.
