Spiders - How Do I Control Them?

[smoknz28]

vB 3.6.1

I don't understand fully about what these 'spiders' are about however, I would like to reduce how many of them hit my site.

Today I had 120+ of them at one time viewing one of my forums. What I do understand is that these can slow down my site.

Is there a way that I can stop these or limit these spiders such as Yahoo! Slurp Spider from hitting my site?

Thanks gang,
Mark

www.f-bodyhideout.com

[letsjoy]

yes u can ban al spiders... but doin that will not ur pages cralwed in SE's

[raiderlax]

how do you get more spiders to go on your site?

[Spiffware]

how do you get more spiders to go on your site?
im with you i think its in the right keywords...

[Exitilus]

You submit your site to the Search Site (ie Google & Yahoo)

[TheBlackPoet]

i have always liked a few yahoo spiders on my site crawling... (at least when i learned they help with SEO's)... but this weekend, i was bombarded with 100 then 150... when i got up to 185 spiders, i knew that was a bit much... so now I'm asking, do i really want that uninvited intrusion on my site? even a good thing should have permission to be good...

my five-minute search lead me to making a robots.txt page..... i collected a few lines of disallow texts to block spiders... i dont know if this will work, but until someone smarter comes up with something better, i havent had a new spider since i uploaded this...