Quote:
Originally Posted by Max Taxable
Robots.txt doesn't disallow or block anything. Think of it as a sign on the door saying "please don't look in the drawers in the rooms." The search engine spiders don't get blocked, and don't have to comply in any way. It's just a way to ask the nice ones "please don't index this."
If you want human guests to see something while not logged in, then google and every other spider can see it as well.
|
Alright thanks.
Then is it possible to do this via .htaccess?