How to block crawl bots without blocking users?

[Max Taxable]

Originally Posted by kai_sey

Would the above disallow also include blocking of the actual threads & posts in that forum?.

Robots.txt doesn't disallow or block anything. Think of it as a sign on the door saying "please don't look in the drawers in the rooms." The search engine spiders don't get blocked, and don't have to comply in any way. It's just a way to ask the nice ones "please don't index this."

If you want human guests to see something while not logged in, then google and every other spider can see it as well.