The Arcive of Official vBulletin Modifications Site.It is not a VB3 engine, just a parsed copy! |
|
|
|||||||
|
#1
05-07-2007, 05:31 PM
|
|||
|
|||
Ayieeee!
Yahoo Slurp Spider has seemingly declared war upon my site.
 This happening to anyone else today? My robots.txt has been fine for ages - I'm wondering if they changed something.
|
|
#2
05-07-2007, 08:23 PM
|
|||
|
|||
|
shoot me your logs, damn, i'd like Yahoo to act like this on my own site...
yeah, yahoo was acting today on some of my servers too... basically because my DNS changed and because some friends searched a lot of keywords found on my sites...
|
|
#3
05-08-2007, 12:30 AM
|
|||
|
|||
|
The hounds are at bay now. I triple-checked my robots.txt, even rewrote it from scratch, but nothing would stop the Mighty Slurp Invasion!
My poor server just threw up the white surrender flag and screamed "No mas!". 
|
|
#4
05-08-2007, 11:29 AM
|
||||
|
||||
|
Quote:
What does your robot.txt file look like? _V
|
|
#5
05-08-2007, 03:14 PM
|
|||
|
|||
|
Yesterday, I submitted my site to yahoo (like I do once a day) and within 2 minutes, I had refreshed my browser and my site went from a normal 34 online for 10 PM at night to 72...over 30 spiders. Made the server nearly crash lol, I was getting messages going "wtf happened to the site". Glad I didn't get hit as hard as your site, Chris
|
|
#6
05-08-2007, 03:49 PM
|
||||
|
||||
|
Try adding this to the very top of your robot.txt file:
User-agent: Slurp Crawl-delay: 20 See if it helps at all, cheers. Edit: Seems 20 is the number even though yahoo says 5-10. http://help.yahoo.com/help/us/ysearc.../slurp-03.html Cheers, _V
|
|
#7
05-08-2007, 03:52 PM
|
|||
|
|||
|
^ My rt looks like his (with a bunch of VBSEO stuff in it as well).
My delay is set at 20 now and it seems to be doing the trick.
|
|
#8
05-08-2007, 03:55 PM
|
|||
|
|||
|
I have mine set to
User-agent: Slurp Crawl-delay: 5 if not there all over
|
 |
«
Previous Thread
|
Next Thread
»
Linear Mode
|
|
All times are GMT. The time now is 04:56 AM.