they crawl all parts of my forum including vb arcade games that i don't have turned on. if you want to prevent it, you can create a robot file in your root to stop them. there are threads on it here. besides if only members can access certain parts that spiders are picking up on, then peeps searching the web will only see your domain and the key words used for search. peeps would have to register to see any members only parts.
|