It's really hard to say why yahoo sends in 5 platoons of troops to crawl a site, it's still a question they would have to answer.
I've encountered that on a few sites i admin for, amending the robots file will stop them, but if i recall its a 30 day waiting period for yahoo to check the file again, an easy and fast way to get rid of them is to ban all but one of yahoo bot ips, i think they use about 30 ips to.
03-23-2008, 12:10 PM
Threaded Mode