vb.org Archive
Page 1 of 3 1 23
Show 40 post(s) from this thread on one page

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vB3 General Discussions (https://vborg.vbsupport.ru/forumdisplay.php?f=111)
-   -   Yahoo! Slurp Spider, Sucking Us Dry (https://vborg.vbsupport.ru/showthread.php?t=102566)

Retell 12-09-2005 01:27 AM
Yahoo! Slurp Spider, Sucking Us Dry
 
I know that spiders are generally good, and about the robot.txt, except I don't want to stop spiders, just have less of them. There is not a single time where we have less than 15 Yahoo! Slurp Spiders on our site. Is there somewhere I can report this? I can understand having 1-3 on once every now and then, but theses guys have been here for more than 2 years!!!


Code:
Yahoo! Slurp Spider  Viewing Thread
Going to be at Quakecon!!!  68.142.250.109 
10:10 PM Yahoo! Slurp Spider  Viewing Thread
Web to Be Major Player at Olympic Games and Euro 2004 (Reuters)  68.142.250.11 
10:23 PM Yahoo! Slurp Spider  Viewing Thread
Thief: Deadly Shadows bug reported  68.142.250.167 
10:18 PM Yahoo! Slurp Spider  Viewing Thread
Cracked Side Panel, what can be done?  68.142.251.154 
10:11 PM Yahoo! Slurp Spider    Viewing User Profile
imbusion  68.142.251.101 
10:24 PM Yahoo! Slurp Spider    Viewing Thread
pics of nzxt's next case  68.142.249.144 
10:13 PM Yahoo! Slurp Spider  Viewing Thread
HDD fan and SYS fan control speed (Nemesis elite)  68.142.251.82 
10:24 PM Yahoo! Slurp Spider  Viewing Thread
Free ipods?  68.142.250.99 
10:23 PM Yahoo! Slurp Spider  Viewing Thread
Future Comp Specs... Good?  68.142.251.163 
10:17 PM Yahoo! Slurp Spider  Viewing Thread
Microsoft to Let Partners Own CE Changes (Reuters)  68.142.249.190 
10:10 PM Yahoo! Slurp Spider    Replying to Thread
Battery dead agian  68.142.249.135 
10:18 PM Yahoo! Slurp Spider  Viewing Thread
Nemesis Elite problems: I love it, but Grrr!  68.142.250.151 
10:12 PM Yahoo! Slurp Spider  Viewing Thread
Star Wraith: Shadows of Orion v1.7 Demo  68.142.249.121 
10:21 PM Yahoo! Slurp Spider  Viewing Thread
Nokia's New N-Gage May Sell Cheaper Than Announced (Reuters)  68.142.251.49 
10:20 PM Yahoo! Slurp Spider    Viewing Thread  68.142.251.43 
10:17 PM Yahoo! Slurp Spider  Viewing Thread
Intel Launches Advanced Notebook PC Processor (Reuters)  68.142.250.74 
10:11 PM Yahoo! Slurp Spider  Viewing Printable Version
Other products  68.142.250.137 
10:24 PM Google Spider  Viewing Thread
My lighted Trinity!  66.249.66.197 
10:19 PM Yahoo! Slurp Spider  Viewing Thread
Guardian Reviews  68.142.250.89 
10:21 PM Yahoo! Slurp Spider  Viewing Thread
my take on the nemesis elite  68.142.251.31

Please Help

Retell :pirate:

Paul M 12-09-2005 01:30 AM
Why don't you want them ?

Most people would kill to have so much search engine attention ....

Guest190829 12-09-2005 01:37 AM
Yes spiders are a good thing, I would let them "slurp" away. :)

Dan 12-09-2005 02:32 AM
I've yet to not see Yahoo Spidering my main site since I reopened it a year ago.

Retell 12-09-2005 08:02 AM
WEll about once a month we somehow end up with 95 slurps... I think that is DEF pushing it... Hehe sorry for stealing all the slurps from you guys :D

Andreas 12-09-2005 08:13 AM
At the moment:
1x Adsense
1x MSN
1x Gigablast (? Don't know that)
8x Google
23x Yahoo!

... less than average.

Guest210212002 11-05-2006 06:12 PM
Pardon the old bump, but I'm getting slurped out myself. I had 81 of them on my site this morning.

ForumDog 11-06-2006 12:27 PM
Yahoo Slurp! (and a few others) generally make a mass invasion around once a month so don't take that to be the norm unless it really is the norm. Otherwise it's a common complaint that Yahoo Slurp! drains more resources than most spiders and there's really nothing you can do aside from blocking them entirely.

Paul M 11-06-2006 02:09 PM
Check our WOL - atm we have around 750 Yahoo spiders online !

4yBak 11-06-2006 04:38 PM
I thinks that Yahoo SLurp! is stupid crawler, because he don't follow with rules in robots.txt. For example: I was restrict access for all crawlers for few files:
Code:
Disallow: /attachment.php
Disallow: /cron.php
Disallow: /newreply.php
Disallow: /newthread.php
and all crawlers are follow with this instructions and don't call this files, but Yahoo Slurp! is still call this files.

Also, few days ago at my board was online ~125 Yahoo Slurp! crawlers (each 2 seconds come new Yahoo Slurp! crawler) - I think it was tempopary bug with crawler :)


All times are GMT. The time now is 05:36 PM.
Page 1 of 3 1 23
Show 40 post(s) from this thread on one page

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01384 seconds
  • Memory Usage 1,735KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (2)bbcode_code_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (2)pagenav_pagelink
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (10)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 
Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 
Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • pagenav_page
  • pagenav_complete
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete