![]() |
Search Bot Increase?
I have noticed lately that the Yahoo Slurp Bots are hitting hard.
When I see them in mass on my forum, sure enough I come here, and they are in mass as well. When they die down, sure enough they are died down here too. Why is that you think? Right now I got over 250: https://vborg.vbsupport.ru/external/2008/03/49.jpg |
Toss this:
Code:
User-agent: Slurp |
Thats the crawl delay, but I do not think it stops the number of spiders at once that crawl.
They need to implement something like: Spider-max: 20 (which would mean a maximum of 20 spiders at a time) It puzzles me too why they never fixed this. |
your avatar is pretty hilarous
|
You should see the full version ;)
|
All times are GMT. The time now is 10:28 AM. |
Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.
X vBulletin 3.8.12 by vBS Debug Information | |
---|---|
|
|
![]() |
|
Template Usage:
Phrase Groups Available:
|
Included Files:
Hooks Called:
|