The Arcive of Official vBulletin Modifications Site.It is not a VB3 engine, just a parsed copy! |
|
#1
|
||||
|
||||
Blocking spiders
I'm not sure why but my forum had a ton of spiders on it at any given time. I had over 3,000 guest on last week and know the majority were spiders. Is there anything I can do to limit them to certain hours?
|
#2
|
||||
|
||||
you can't really pick the time, but chances are good most were the Yahoo spiders. You can place a directive in your robots.txt with a crawl delay for them that will slow them down.
|
#3
|
||||
|
||||
How? I don't believe I have a robots.txt file. Where would that be?
|
#4
|
||||
|
||||
its in your website root ex:
http://seotips4.com/robots.txt http://vbulletin.org/robots.txt Its never in the forum root. Notice the Code:
User-agent: Slurp Crawl-delay: 5 |
#5
|
||||
|
||||
Many thanks. I'll give it a go.
|
|
|
X vBulletin 3.8.12 by vBS Debug Information | |
---|---|
|
|
More Information | |
Template Usage:
Phrase Groups Available:
|
Included Files:
Hooks Called:
|