So why has my site become popular for spiders, webcralwlers and bots now?

[Southernphuk]

Hola,

I've noticed over the last month that the spiders, bots and webcrawlers have really punched it up a notch, where I used to get around 150-200 a day I'm now getting closer to 2000 of them a day now and I can't for the life of me figure out why!

Any clues or ideas for the clueless here? I've got a robot.txt file but that is to keep them out of certain areas that they really have no business bothering with (profiles, pm's, etc) but have shied away from outright cutting them off from the site as I figure that is a good way to have a presence in the search engines which of course everyone wants them to funnel their way.

Anyhow, extremely curious about this and wouldn't mind some thoughts and insight on this.

[SCRIPT3R]

just count yourself lucky and leave it at that.

[Southernphuk]

Hah, believe me, I really don't want to get rid of them as frankly they aren't eating up 'that' much bandwidth (well, the google one ate through a goodly bit last month but still), I'm just curious what triggers something like this.

Regards,
Nathan

[SCRIPT3R]

nobody really knows for sure... luck of the draw.

[acertek]

Yeah I've noticed this lately too. What I've been noticing is inktomi/yahoo seems to have a spider that appears as 100+ users on your forum and crawls it with multiple legs. I think as your forum grows they put largers spiders on it to crawl it faster.

[Brandon Sheley]

back links
Whats your site url ? I'll be able to tell you in a matter on minutes..

[mfyvie]

Originally Posted by Southernphuk

Any clues or ideas for the clueless here? I've got a robot.txt file but that is to keep them out of certain areas that they really have no business bothering with (profiles, pm's, etc) but have shied away from outright cutting them off from the site as I figure that is a good way to have a presence in the search engines which of course everyone wants them to funnel their way.

Anyhow, extremely curious about this and wouldn't mind some thoughts and insight on this.

Most of the evil ones won't respect robots.txt anyway. You should care about them though - the nastier ones are just stealing your content and bringing you nothing in return. At worst they can bring your site to a standstill if they are poorly written (I've had this happen to me on a few occassions). I block a big list of the bad ones at my webserver and my forum response time increased immediately. I still allow all search engines through unless they are specifically blocked (I think I block around 200).

I also wrote a small mod to remove spiders from the statistics on your forum. This doesn't block anything, but it is useful to get rid of the 100's of sessions started by spiders like Yahoo's slurp.