Snakes thanks for all your replies, I am impressed that you would take the time to answer these posts...

Mike.

your welcome and good luck

I am fed up with 50 or 60 spiders on my site using all my sql queries,

at any time, day or night I have 50-60 spiders per hour...

is there a way to limit how many spiders are crawling all over my site...

Use a robots.txt to block all spiders from crawling your site. You won't get listed in the search engines but it'll stop your query usage. Or at the very least use a robots.txt to limit the crawl rates

But i'd suggest moving to a different host, such as ASmallOrange.com or PowerVPS if your site warrants a VPS.

My question really is why do the spiders need to crawl it so often, of course we all want to be found in the search engines but really 50-60 spiders every hour is a bit strange, is it normal, am I have less or more than others?

I have a small non profit site that I am trying to grow and I don't want all my sql queries being used by spiders and yet I still want to be found by the search engines, i would love a vps but couldn't afford to sustain one... I have reseller hosting with Hostgator...

Google will usually only send one spider thats really efficent, its probably that pesky yahoo bot which is killing you.

Use robots.txt to slow the spiders, you can see what should be in a standard vBulletin robots.txt and where to put it on your server, and then just adding something like

to the end of it will slow the bots. These changes probably won't take effect for 24hrs, as most spiders only download robots.txt file once a day.

Wow thanks for that...

is the crawl delay value in minutes like 120 minutes?

thanks again.. Mike.

--------------- Added [DATE]1207078601[/DATE] at [TIME]1207078601[/TIME] ---------------

worked a charm, I have only 8 spiders on my site now... thanks for the tip...

Well no.

Personally I would much prefer a million spiders and spend a bit more to be on a better server. Anyway to ensure all my content is being indexed at a faster speed in the hope to attract more members for the content that i'm trying to drive members to is important IMO.

As for a good host, there are many, it does not seem like your forums are that busy or large currently and you can still get away with shared hosting until you start to get quite a lot larger where you start using too many resources for that server/host where it would become unfair to the other people on the server.
Some hosts are good with their tos in this respect, others have things in place where you use x amount cpu you will be suspended and the value is not even that high sometimes.

I run a hosting company (wont release the site here) but I feel if a hosting company is ever suspending or terminating sites without giving adequate warning and advise before hand, then you should never go with that host.

Remember in most cases you are paying for what you get. In some cases you may find a golden treasure but for just how long will it last? Take a look at how long they have been around, look for reviews etc. WHT can be a good place to start looking for hosts as well.

I wish you luck in finding a good host :)