Quote:
Originally Posted by Lee G
Dont know if its listed in your bad user agent list, one of the worst I have found is OMGILI
User agent
omgilibot/0.3 + http://www.omgili.com/Crawler.html
If you right click on any page on the search engine site, you will see that all external links are no follow
They were hitting me for over 200 pages a day until I put an ip ban on them
|
Hmm, it's not added there but folk can tailor their own, i only gave a starting base, i'm sure many have now customised it
I can bet that omgili aren't as bad as Baidu