|
What is this?
This script will create a Robot Exclusion
file from the information you provide. Why should I use it?
Often there are directories in your Website that you do not want to be indexed
e.g. CGI-BIN. This file will be read by any indexing spiders and those that are
legitimate will then not index any files within the directories you wish to keep
private.
Warning
This file can if used improperly deny access to all search engine spiders. If
you do not know what 'User Agents' are then please leave it set to 'All Agents
Default'
Instructions
To start simply add a directory per line into the form below that you wish
to keep way from the indexing spiders and click on 'Create Robot File'.
|
