Blocking crawlers from certain pages (htaccess)?

[DisasterDotCom]

Hi All -

Has anyone put an htaccess together to block crawlers from indexing the following aspects of vb5?


Profiles
Signature invisible for guests/google bot
Print pages
Stats
Help
Attach
Subscribe
Search

[ozzy47]

You would want to use a robots.txt file.
https://support.google.com/webmaster.../6062596?hl=en (https://support.google.com/webmasters/answer/6062596?hl=en)
https://support.google.com/webmaster.../6062608?hl=en (https://support.google.com/webmasters/answer/6062608?hl=en)

All good spiders will follow this, but some of the bad ones might not.

[Max Taxable]

Keep in mind that doesn't block anything, it's just a "please don't index these" kind of deal and it is a PUBLICLY READABLE document. Meaning your admincp and modcp locations will be PUBLISHED.

Use "ban Spiders by User Agent" mod instead, for the bad actors.

Hi All -

Has anyone put an htaccess together to block crawlers from indexing the following aspects of vb5?


Profiles
Signature invisible for guests/google bot
Print pages
Stats
Help
Attach
Subscribe
Search
Plus these are all blockable from public view, in Usergroup Options.

[ozzy47]

He can not use those mods, as he has a vB5 site.

[Max Taxable]

He can not use those mods, as he has a vB5 site.Yeah I know - wanna show him what he's missing without hooks!:D