Search bots, whats up with them???

[Rick Sample]

I'm not sure which engines I have my site listed on. I enabled the "show bots in who's online" When google searches and stuff, it shows google bot instead of just guest

I have just the default ones listed their. How can I get all of them listed their that are cruising my site? I know its the bots because when you have like 20 ip address or something close and they all are very similar to: 64.12.117.22 just the last two numbers are changed. Does anyone know what bot this is so I can add it to show which bot in who's online?

[mmw]

You should check out http://www.vbulletin.com/forum/showt...ht=search+bots
has lots of info on most of the bots

[Rick Sample]

Thanks man, I just added them all I hope this will help a little

[Mudusa]

I had my first batch of google spiders hit the site the other day. Two were searching indexes, the other 20 were going through user profiles and showed activity as "Emailing Member". Thank god on the next refresh it showed them viewing Forbidden message. I set the spiders the other day, sure am glad the emails are hidden, with no permissions set to send them.

[K33nny]

Just use an .htaccess file to block spiders from accessing certain files. I've had MSN, Ask Jeeves, Google, Inktomi, Alexa, and a few random's pounding my website this past week. They're only allowed to access the index, link database, forumdisplay.php, showthread.php, and the archive.

[Fluous]

Funny thing is. The top number of users online I had ever hit was 20 (Mind you I've only been open like a week and a half.) And that was the first day. Then it dropped down to like 2-4. Suddenly, I looked, and it says Most Users Online ever was 115. Like two days ago. The only thing that could have happened was that probably around 112 bots hit at the same time. Earlier I saw 2 Yahoo Slurp bots. When I get more activity, and my archive working I should get more. But I don't understand how they all came in at once. I have set up a couple of google leaderboards on a html file and placed it in every single directory. Maybe that attracted them? The google html files aren't anywhere placed on the site, just random html in a directory.

[Rick Sample]

Originally Posted by Mudusa

I had my first batch of google spiders hit the site the other day. Two were searching indexes, the other 20 were going through user profiles and showed activity as "Emailing Member". Thank god on the next refresh it showed them viewing Forbidden message. I set the spiders the other day, sure am glad the emails are hidden, with no permissions set to send them.

Yeah, when bots come to my site I see them tring to e-mail members, PM members, login, and I have even had some bots try to register. LOL...they must of liked my site.

Anywase, they can't send e-mails or anything that guests can't do. Because they are just like guests. Their not registered, therefore they will get the forbiden message when the try to access those pages. I like when bots visit though. it makes my member base go up, (since we're a new site) therefore, when real guests visit their more likely to sign up if they see 20 users online, even though their just bots

[twinsmom]

You can also restrict their access by using a robots.txt file in your root directory.

http://www.robotstxt.org/wc/robots.html

[filburt1]

Never ever ever block access to search engines unless it is 100% absolutely necessary. You're shooting yourself in the foot or (nameless body part) otherwise.

[twinsmom]

I agree that totally disallowing them would be shooting yourself in the foot, but there needs to be a balance between having them around and restricting their access somewhat. Not letting them go through 'endless loops' of things like sending a PM or contact email, or replying to a post makes sense and potentially helps save bandwith doesn't it? I'm not sure how that would be a bad thing?

Of course, I'm a newbie at this so maybe you know something I don't?