Go Back   vb.org Archive > vBulletin 3 Discussion > vB3 General Discussions
Reload this Page Yahoo! Slurp Spider, Sucking Us Dry
FAQ Community Calendar Today's Posts Search

Reply
Page 1 of 3 1 23
 
Thread Tools Display Modes
  #1  
Old 12-09-2005, 01:27 AM
Retell's Avatar
Retell Retell is offline
 
Join Date: Jul 2005
Location: *Cough*New Jersey*Cough*
Posts: 70
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default Yahoo! Slurp Spider, Sucking Us Dry
I know that spiders are generally good, and about the robot.txt, except I don't want to stop spiders, just have less of them. There is not a single time where we have less than 15 Yahoo! Slurp Spiders on our site. Is there somewhere I can report this? I can understand having 1-3 on once every now and then, but theses guys have been here for more than 2 years!!!


Code:
Yahoo! Slurp Spider   Viewing Thread 
Going to be at Quakecon!!!  68.142.250.109  
10:10 PM Yahoo! Slurp Spider   Viewing Thread 
Web to Be Major Player at Olympic Games and Euro 2004 (Reuters)  68.142.250.11  
10:23 PM Yahoo! Slurp Spider   Viewing Thread 
Thief: Deadly Shadows bug reported  68.142.250.167  
10:18 PM Yahoo! Slurp Spider   Viewing Thread 
Cracked Side Panel, what can be done?  68.142.251.154  
10:11 PM Yahoo! Slurp Spider    Viewing User Profile 
imbusion  68.142.251.101  
10:24 PM Yahoo! Slurp Spider    Viewing Thread 
pics of nzxt's next case  68.142.249.144  
10:13 PM Yahoo! Slurp Spider   Viewing Thread 
HDD fan and SYS fan control speed (Nemesis elite)  68.142.251.82  
10:24 PM Yahoo! Slurp Spider   Viewing Thread 
Free ipods?  68.142.250.99  
10:23 PM Yahoo! Slurp Spider   Viewing Thread 
Future Comp Specs... Good?  68.142.251.163  
10:17 PM Yahoo! Slurp Spider   Viewing Thread 
Microsoft to Let Partners Own CE Changes (Reuters)  68.142.249.190  
10:10 PM Yahoo! Slurp Spider    Replying to Thread 
Battery dead agian  68.142.249.135  
10:18 PM Yahoo! Slurp Spider   Viewing Thread 
Nemesis Elite problems: I love it, but Grrr!  68.142.250.151  
10:12 PM Yahoo! Slurp Spider   Viewing Thread 
Star Wraith: Shadows of Orion v1.7 Demo  68.142.249.121  
10:21 PM Yahoo! Slurp Spider   Viewing Thread 
Nokia's New N-Gage May Sell Cheaper Than Announced (Reuters)  68.142.251.49  
10:20 PM Yahoo! Slurp Spider    Viewing Thread  68.142.251.43  
10:17 PM Yahoo! Slurp Spider   Viewing Thread 
Intel Launches Advanced Notebook PC Processor (Reuters)  68.142.250.74  
10:11 PM Yahoo! Slurp Spider   Viewing Printable Version 
Other products  68.142.250.137  
10:24 PM Google Spider   Viewing Thread 
My lighted Trinity!  66.249.66.197  
10:19 PM Yahoo! Slurp Spider   Viewing Thread 
Guardian Reviews  68.142.250.89  
10:21 PM Yahoo! Slurp Spider   Viewing Thread 
my take on the nemesis elite  68.142.251.31

Please Help

Retell irate:
Reply With Quote
  #2  
Old 12-09-2005, 01:30 AM
Paul M's Avatar
Paul M Paul M is offline
 
Join Date: Sep 2004
Location: Nottingham, UK
Posts: 23,748
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Why don't you want them ?

Most people would kill to have so much search engine attention ....
Reply With Quote
  #3  
Old 12-09-2005, 01:37 AM
Guest190829
Guest
  
Posts: n/a
Default
Yes spiders are a good thing, I would let them "slurp" away.
Reply With Quote
  #4  
Old 12-09-2005, 02:32 AM
Dan's Avatar
Dan Dan is offline
 
Join Date: Dec 2002
Location: Titusville, Florida
Posts: 1,787
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
I've yet to not see Yahoo Spidering my main site since I reopened it a year ago.
Reply With Quote
  #5  
Old 12-09-2005, 08:02 AM
Retell's Avatar
Retell Retell is offline
 
Join Date: Jul 2005
Location: *Cough*New Jersey*Cough*
Posts: 70
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
WEll about once a month we somehow end up with 95 slurps... I think that is DEF pushing it... Hehe sorry for stealing all the slurps from you guys
Reply With Quote
  #6  
Old 12-09-2005, 08:13 AM
Andreas's Avatar
Andreas Andreas is offline
 
Join Date: Jan 2004
Location: Germany
Posts: 6,863
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
At the moment:
1x Adsense
1x MSN
1x Gigablast (? Don't know that)
8x Google
23x Yahoo!

... less than average.
Reply With Quote
  #7  
Old 11-05-2006, 06:12 PM
Guest210212002
Guest
  
Posts: n/a
Default
Pardon the old bump, but I'm getting slurped out myself. I had 81 of them on my site this morning.
Reply With Quote
  #8  
Old 11-06-2006, 12:27 PM
ForumDog ForumDog is offline
 
Join Date: Jul 2006
Posts: 84
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Yahoo Slurp! (and a few others) generally make a mass invasion around once a month so don't take that to be the norm unless it really is the norm. Otherwise it's a common complaint that Yahoo Slurp! drains more resources than most spiders and there's really nothing you can do aside from blocking them entirely.
Reply With Quote
  #9  
Old 11-06-2006, 02:09 PM
Paul M's Avatar
Paul M Paul M is offline
 
Join Date: Sep 2004
Location: Nottingham, UK
Posts: 23,748
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Check our WOL - atm we have around 750 Yahoo spiders online !
Reply With Quote
  #10  
Old 11-06-2006, 04:38 PM
4yBak 4yBak is offline
 
Join Date: Apr 2004
Posts: 78
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
I thinks that Yahoo SLurp! is stupid crawler, because he don't follow with rules in robots.txt. For example: I was restrict access for all crawlers for few files:
Code:
Disallow: /attachment.php
Disallow: /cron.php
Disallow: /newreply.php
Disallow: /newthread.php
and all crawlers are follow with this instructions and don't call this files, but Yahoo Slurp! is still call this files.

Also, few days ago at my board was online ~125 Yahoo Slurp! crawlers (each 2 seconds come new Yahoo Slurp! crawler) - I think it was tempopary bug with crawler
Reply With Quote
Reply
Page 1 of 3 1 23

« Previous Thread | Next Thread »

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT. The time now is 12:50 PM.

Contact Us - Archive - Top

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.
X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.04241 seconds
  • Memory Usage 2,246KB
  • Queries Executed 11 (?)
More Information
Template Usage:
  • (1)SHOWTHREAD
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)ad_showthread_beforeqr
  • (1)ad_showthread_firstpost
  • (1)ad_showthread_firstpost_sig
  • (1)ad_showthread_firstpost_start
  • (2)bbcode_code
  • (1)footer
  • (1)forumjump
  • (1)forumrules
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (1)navbar
  • (3)navbar_link
  • (120)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (2)pagenav_pagelink
  • (10)post_thanks_box
  • (10)post_thanks_button
  • (1)post_thanks_javascript
  • (1)post_thanks_navbar_search
  • (10)post_thanks_postbit_info
  • (10)postbit
  • (8)postbit_onlinestatus
  • (10)postbit_wrapper
  • (1)spacer_close
  • (1)spacer_open
  • (1)tagbit_wrapper 
Phrase Groups Available:
  • global
  • inlinemod
  • postbit
  • posting
  • reputationlevel
  • showthread
Included Files:
  • ./showthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/functions_bigthree.php
  • ./includes/class_postbit.php
  • ./includes/class_bbcode.php
  • ./includes/functions_reputation.php
  • ./includes/functions_post_thanks.php 
Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • showthread_start
  • showthread_getinfo
  • forumjump
  • showthread_post_start
  • showthread_query_postids
  • showthread_query
  • bbcode_fetch_tags
  • bbcode_create
  • showthread_postbit_create
  • postbit_factory
  • postbit_display_start
  • post_thanks_function_post_thanks_off_start
  • post_thanks_function_post_thanks_off_end
  • post_thanks_function_fetch_thanks_start
  • post_thanks_function_fetch_thanks_end
  • post_thanks_function_thanked_already_start
  • post_thanks_function_thanked_already_end
  • fetch_musername
  • postbit_imicons
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • postbit_display_complete
  • post_thanks_function_can_thank_this_post_start
  • pagenav_page
  • pagenav_complete
  • tag_fetchbit_complete
  • forumrules
  • navbits
  • navbits_complete
  • showthread_complete