To block the crawlers go to certain pages, create a robots.txt file with the content:
Code:
User-agent: *
Disallow: /forum/admin/
Disallow: /forum/images/
Disallow: /forum/mod/
Disallow: /forum/attachment.php
Disallow: /forum/avatar.php
Disallow: /forum/editpost.php
Disallow: /forum/member.php
Disallow: /forum/member2.php
Disallow: /forum/memberlist.php
Disallow: /forum/misc.php
Disallow: /forum/moderator.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/poll.php
Disallow: /forum/postings.php
Disallow: /forum/printarticle.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/private2.php
Disallow: /forum/register.php
Disallow: /forum/report.php
Disallow: /forum/search.php
Disallow: /forum/sendtofriend.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
and upload it onto your
main (not /forum, the one above it) directory. Crawlers will look ONLY onto the root folder for the robots.txt file.
Make sure you change the /forum folder to your actual folder name.
For more info on robots.txt file, go
here.