Filed in archive
google
by Arun Radhakrishnan on March 30, 2008
Google has released a tool at Google Webmaster Central that lets users generate robot.txt files for their sites. A robot.txt file is like is what the spiders or crawlers of search engines read to know what part of the website they have access to. There is a general introduction to the robot.txt file at SearchEngineLand ( also the source of the information). A general know how on the format of robot.txt is necessary if you want to maintain you own website.
Things to keep in mind are that the robot.txt file must lie in the root directory of the website and also that you have to get the names of the specific spiders or crawlers or bots of searchengines right if you have to address them to make certain parts of your site unaccessible.
Things to keep in mind are that the robot.txt file must lie in the root directory of the website and also that you have to get the names of the specific spiders or crawlers or bots of searchengines right if you have to address them to make certain parts of your site unaccessible.
Permalink: Robot.txt Generator From Google
Trackback: http://publish.creative-weblogging.com/publish/mt-tb.pl/118533
Addthis
Ask
Blinklist
del.icio.us
Digg
Fark
Facebook
Google
Lycos
Ma.gnolia
Mr Wong
Netscape
Netvousz
Newsvine
Reddit
StumbleUpon
Slashdot
Tailrank
Technorati
Wink
Yahoo
Vote for Robot.txt Generator From Google:
|
Rating: 8.00 out of 1 vote(s) cast.
|
Response from:
Madan
(08/14/08 11:47pm)
robot
Subscribe
Use the search to look for other interesting posts
| RSS |  See all blog subscribe options |
 What is RSS? | |
| Yahoo! |
|
| Addthis |
|
| Bloglines |
|
| Newsletter | |
| Follow us on Twitter! |
