How to block crawl bots without blocking users?
 

I need to block google & co from searching particular forums on my site, whilst still allowing access to guests & members.

I know this can be done via the robots.txt file - I'm just wondering what to put there.

Lets say the particular forum I want to block is: www.domain.com/forums/audi-versus-bmw/ (Note: I need to disallow all threads/posts in that particular forum).

So, is the following correct?

User-agent: *
Disallow: /forums/audi-versus-bmw/

Would the above disallow also include blocking of the actual threads & posts in that forum? Or would it only block the actual url www.domain.com/forums/audi-versus-bmw?

Thanks.

Robots.txt doesn't disallow or block anything. Think of it as a sign on the door saying "please don't look in the drawers in the rooms." The search engine spiders don't get blocked, and don't have to comply in any way. It's just a way to ask the nice ones "please don't index this."

If you want human guests to see something while not logged in, then google and every other spider can see it as well.

Alright thanks.

Then is it possible to do this via .htaccess?

Yes, you can use this list for example.

http://tab-studio.com/en/blocking-robots-on-your-page/