The Arcive of Official vBulletin Modifications Site.It is not a VB3 engine, just a parsed copy! |
|
#1
|
||||
|
||||
![]()
Hola,
I've noticed over the last month that the spiders, bots and webcrawlers have really punched it up a notch, where I used to get around 150-200 a day I'm now getting closer to 2000 of them a day now and I can't for the life of me figure out why! Any clues or ideas for the clueless here? I've got a robot.txt file but that is to keep them out of certain areas that they really have no business bothering with (profiles, pm's, etc) but have shied away from outright cutting them off from the site as I figure that is a good way to have a presence in the search engines which of course everyone wants them to funnel their way. Anyhow, extremely curious about this and wouldn't mind some thoughts and insight on this. |
#2
|
|||
|
|||
![]()
just count yourself lucky and leave it at that.
|
#3
|
||||
|
||||
![]()
Hah, believe me, I really don't want to get rid of them as frankly they aren't eating up 'that' much bandwidth (well, the google one ate through a goodly bit last month but still), I'm just curious what triggers something like this.
Regards, Nathan |
#4
|
|||
|
|||
![]()
nobody really knows for sure... luck of the draw.
|
#5
|
|||
|
|||
![]()
Yeah I've noticed this lately too. What I've been noticing is inktomi/yahoo seems to have a spider that appears as 100+ users on your forum and crawls it with multiple legs. I think as your forum grows they put largers spiders on it to crawl it faster.
|
#6
|
||||
|
||||
![]()
back links
![]() Whats your site url ? I'll be able to tell you in a matter on minutes.. |
#7
|
|||
|
|||
![]() Quote:
I also wrote a small mod to remove spiders from the statistics on your forum. This doesn't block anything, but it is useful to get rid of the 100's of sessions started by spiders like Yahoo's slurp. |
![]() |
Thread Tools | |
Display Modes | |
|
|
X vBulletin 3.8.12 by vBS Debug Information | |
---|---|
|
|
![]() |
|
Template Usage:
Phrase Groups Available:
|
Included Files:
Hooks Called:
|