Some more commands on Robots.txt

Write a Robots.txt File - Writing the File, Using Disallows

The second line known as the directive is written as:
Disallow:

By adding a folder after the Disallow statement, the search spider
should ignore the folder for indexing purposes and move to others where there is
no restriction.

Disallow: /images/

This is a special example, just for Perfect 10. This one minute
bit of instruction could have saved a ton in wasted legal fees on a
frivolous lawsuit. As this is a basic step in building websites, it is incumbent
on website owners to their intellectual property, and not a 3rd party
search engines duty.

You can also disallow specific files this way

Disallow:cheeseyporn.htm

One way I recommend using this all the time is to keep robots out of
you cgi bin directory

Disallow: /cgi-bin/

If you leave the Disallow
directive line blank or not filled in, this indicates that ALL files may be retrieved and or indexed by specifiedl robot(s).This would let all robots index all files.
User-agent: *

Disallow:

And vice versa you can keep all robots out easily.
User-agent: *

Disallow: /

In the example above, the one forward slash (/) equals your root
directory. Since the root directory is blocked, none of the other folders and
files can be indexed or crawled. Your site will be removed from search engines
once they read your robots.txt and update their indexes.

---

---