|
Creating
Your Robots.txt File
You should not create your
robots.txt file in your web creator program, but create it in either NotePad
or TextPad. It should be saved as robots.txt (NOT with an .html or .htm
extension). It will then be uploaded to the root directory, so it will
appear as http://www.example.com/robots.txt
Allowing all bots
Your basic robots.txt file (what you likely want to use if you haven't
been using one until now) will allow all bots to index all pages of your
site. The * is generic to refer to all bots.
Disallowing all bots
Your basic robots.txt file would not allow any bot to access any page
on your site they can find.
User-agent:
*
Disallow: / |
Disallowing specific bots
If you would like to not allow certain bots to index your pages, you can
refer to bots by name and disallow them. For example, many webmasters
prefer not to allow Google's Image Search bot to index any of the images
on their site.
User-agent:
Googlebot-Image
Disallow: / |
Disallowing Specific Pages
& Directories
You may want to not allow bots to index certain pages or directories on
your site. Do keep in mind that if you have any private directories that
are listed in your robots.txt that you should password protect them. Otherwise
you have just alerted rogue bots (that ignore contents of your robots.txt
file or specifically target disallowed pages and directories) as well
as any competitors to your private directories. In these cases, you should
disallow robots using meta tags instead.
cgi-bin is a common directory
to disallow, since it normally contains scripts, many that don't need
indexing and which can go through a lot of bandwidth as a result.
User-agent:
*
Disallow: /cgi-bin/
Disallow: /privatedirectory/
Disallow: /directory/largepage.htm |
Once it is completed, save
it as robots.txt and upload it to your server.
Alternatively, you can use
our free robots.txt generator to have
us create your robots.txt generator for you.
|
