Disallow certain types of files in robots.txt?

[iamboard]

Is it a good idea to disallow search engines from indexing some types vbulletin files...such as http://www.mysite.com/forum/member.php?u=1120? Any other types?

What do you write in the robots.txt file?

[daloosaa]

Hi,

yes its a very good idea, because you can restrict indexing of files with no content such as report.php or register.php. But note: a robots.txt is no guarantee that search engines doesn't spider these files.

Greetings,
Florian