Htaccess deny folder access

[SVTCobraLTD]

I keep getting the following error showing in my cpanel

[Fri May 14 07:49:19 2010] [error] [client xxx.xx.xx.xxx] File does not exist: /home/xxxxxx/public_html/forums/photoalbum

How can I block it so spiders are not looking for this photoalbum folder? I am pretty sure all links are removed for that but for some reason, spiders keep looking for it.

[vbenhancer]

spiders are not "visiting" each page of your site, they are crawling and storing the info in their cache... so when they return, they validate their cache by visiting back the pages they crawled once...

so yes, they hit the same files even if they do not exist anymore.

you need to contact your webhost to see if you can learn from them how to manage these details inside your CPanel...

[SVTCobraLTD]

So there is no htaccess setting I can use to block them trying to crawl that section?

[USAMustangs.com]

Just add an exclusion to your robots.txt file

Example:


User-agent: *
Disallow: /forum/admincp/
Disallow: /forum/modcp/
Disallow: /forums/photoalbum/

forums or forum....how ever you have your forum directory labeled.

And in your .htaccess file in the photoalbum directory, you can insert the following syntax to protect the .htaccess file itself from the outside world.

<Files .htaccess>
order deny,allow
deny from all
</Files>

[SVTCobraLTD]

Thank you. That was a big help.