vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   Community Lounge (https://vborg.vbsupport.ru/forumdisplay.php?f=13)
-   -   Search bots, whats up with them??? (https://vborg.vbsupport.ru/showthread.php?t=64814)

Rick Sample 05-06-2004 09:23 PM

Search bots, whats up with them???
 
I'm not sure which engines I have my site listed on. I enabled the "show bots in who's online" When google searches and stuff, it shows google bot instead of just guest

I have just the default ones listed their. How can I get all of them listed their that are cruising my site? I know its the bots because when you have like 20 ip address or something close and they all are very similar to: 64.12.117.22 just the last two numbers are changed. Does anyone know what bot this is so I can add it to show which bot in who's online?

mmw 05-06-2004 09:49 PM

You should check out http://www.vbulletin.com/forum/showt...ht=search+bots
has lots of info on most of the bots :)

Rick Sample 05-06-2004 10:01 PM

Thanks man, I just added them all I hope this will help a little

Mudusa 05-07-2004 11:30 PM

I had my first batch of google spiders hit the site the other day. Two were searching indexes, the other 20 were going through user profiles and showed activity as "Emailing Member". Thank god on the next refresh it showed them viewing Forbidden message. I set the spiders the other day, sure am glad the emails are hidden, with no permissions set to send them.

K33nny 05-08-2004 12:28 AM

Just use an .htaccess file to block spiders from accessing certain files. I've had MSN, Ask Jeeves, Google, Inktomi, Alexa, and a few random's pounding my website this past week. They're only allowed to access the index, link database, forumdisplay.php, showthread.php, and the archive.

Fluous 05-08-2004 02:42 AM

Funny thing is. The top number of users online I had ever hit was 20 (Mind you I've only been open like a week and a half.) And that was the first day. Then it dropped down to like 2-4. Suddenly, I looked, and it says Most Users Online ever was 115. Like two days ago. The only thing that could have happened was that probably around 112 bots hit at the same time. Earlier I saw 2 Yahoo Slurp bots. When I get more activity, and my archive working I should get more. But I don't understand how they all came in at once. I have set up a couple of google leaderboards on a html file and placed it in every single directory. Maybe that attracted them? The google html files aren't anywhere placed on the site, just random html in a directory.

Rick Sample 05-08-2004 02:54 AM

Quote:

Originally Posted by Mudusa
I had my first batch of google spiders hit the site the other day. Two were searching indexes, the other 20 were going through user profiles and showed activity as "Emailing Member". Thank god on the next refresh it showed them viewing Forbidden message. I set the spiders the other day, sure am glad the emails are hidden, with no permissions set to send them.


Yeah, when bots come to my site I see them tring to e-mail members, PM members, login, and I have even had some bots try to register. LOL...they must of liked my site. :)

Anywase, they can't send e-mails or anything that guests can't do. Because they are just like guests. Their not registered, therefore they will get the forbiden message when the try to access those pages. I like when bots visit though. it makes my member base go up, (since we're a new site) therefore, when real guests visit their more likely to sign up if they see 20 users online, even though their just bots :)

twinsmom 04-07-2005 04:11 AM

You can also restrict their access by using a robots.txt file in your root directory.

http://www.robotstxt.org/wc/robots.html

filburt1 04-07-2005 05:16 AM

Never ever ever block access to search engines unless it is 100% absolutely necessary. You're shooting yourself in the foot or (nameless body part) otherwise.

twinsmom 04-07-2005 01:11 PM

I agree that totally disallowing them would be shooting yourself in the foot, but there needs to be a balance between having them around and restricting their access somewhat. Not letting them go through 'endless loops' of things like sending a PM or contact email, or replying to a post makes sense and potentially helps save bandwith doesn't it? I'm not sure how that would be a bad thing?

Of course, I'm a newbie at this so maybe you know something I don't? :)


All times are GMT. The time now is 08:31 AM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01199 seconds
  • Memory Usage 1,738KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (1)pagenav_pagelink
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (10)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 

Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • pagenav_page
  • pagenav_complete
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete