[NickCat]

Originally Posted by IvyKeepMommy

Well, you could set it so that only registered members can see the chatbox... that would solve the spider issue. It is doesn't, you can use the set it so that only members with X posts can see/use the chatbox.

On our forum, you have to have 50 posts to see/use the chatbox to make new members familiar with the rest of the forum before they get sucked into chatting.

I have it set so that not logged in users can see it, but not post of course. And I do have it set to 10 posts or greater to view, but I'm currently only using the ChatBox Full, since I can't find an easy way to only have it inline in one specific forum on the site, not sitewide. I guess adding not logged in people to the list can't really cause that much harm since if they aren't a user they can't use it anyway.

The root of the problem here is Slurp, it's just completely ignoring the robots.txt file. I found more than a few instances of this while googleing for more info. So I'm not the only one this is happening to. People are reporting that Yahoo is crawling their site, but the stuff it shouldn't be searching never seems to make it into the search engine, however that's still breaking the robots.txt rules, even though Yahoo is adamant they obey them 100%.