Google has released a tool at Google Webmaster Central that lets users generate robot.txt files for their sites. A robot.txt file is like is what the spiders or crawlers of search engines read to know what part of the website they have access to. There is a general introduction to the robot.txt file at SearchEngineLand ( also the source of the information). A general know how on the format of robot.txt is necessary if you want to maintain you own website.

Things to keep in mind are that the robot.txt file must lie in the root directory of the website and also that you have to get the names of the specific spiders or crawlers or bots of searchengines right if you have to address them to make certain parts of your site unaccessible.