Quote:
Originally Posted by Shamil.
If the software blocks the bot at software level, robots.txt won't help you. I still suggest users use Google Webmaster Tools to monitor.
|
I'm sorry but I was responding to a question on what happens if this blocks google search bot.. If someones goal was to block search bots I would use another method but I cannot see why most sites would want to keep from getting indexed.