Nice robots.txt, though it seems a little out of date and missing some files like tags.php albums.php and groups.php you can even take it a little further and block off your entire includes folder (I've seen vBulletin javascript get indexed :roll
and the clientscript folders if you don't want your images cralwed, also block that off
Things to remember:
- Robots.txt can only be served from your public_html (or www) folder. It can't not be placed in a sub-folder on your domain.
- If your vBulletin isn't in /forums be sure to update the path to the correct location or else it won't do any good.
06-24-2009, 06:03 AM
Threaded Mode