Quote:
Originally Posted by Simon Lloyd
i ban the more agressive ones like Baidu, why on earth would they need 215 bots indexing my site?
|
Yeah, 215 Chinese bots leeching resources, to allegedly index your site for people who most likely can never see it. Most US sites are blocked in China. Baidu makes no sense at all, it certainly doesn't help anybody. By far the worst behaving bot out there, it totally ignores robots.txt.