Yet another ROBOTS.txt file question (sorry)

[invitezone]

Can someone give me some advice on this please.
I have a public section and a private section of my forum.
I want to allow bots from search engines to crawl and index my public forum in order to get some traffic through it.
But I also want to absolutely deny any bots at all from access and crawling anything else.

Can I specify a robots.txt that says allow 1 forum category and disallow everywhere else?

I have searched but can't find any information on these specifics. Obviously I found lots of robots.txt stuff, but nothing that can be of specific help.

Thanks for your help.

[Zachery]

Nope, that's not what robots.txt is for anyway.

[invitezone]

ok thanks, so how can I achieve what I described?
any help on that?

Thanks

[nhawk]

ACP->Forums & Moderators->Forum Permissions

Change the 'Unregistered/Not Logged In' permissions for forums you don't want viewed to all 'No'.

With that setting, the forums won't even exist to robots or unregistered users.

[invitezone]

oh as simple as that, ok didn't realise, im still new to most of this. So what is the point of the robots.txt other than to deny all bots? THANKS

[nhawk]

Robots.txt is good for asking robots not to access things like register.php, search.php, etc.

Note I said 'asking'. Not all robots obey robots.txt.

[invitezone]

for what reason would you do that? BTW thanks again for your patience and help

[nhawk]

Many pages serve no purpose to a robot and are just a waste of bandwidth when they crawl those pages.

Register.php, login.php, search.php, subscription.php, profile.php are just a few of those types of pages.

[Macsee]

There are threads around with lists of the pages you might want to include in your robots.txt block.

Note that the blocking suggested above blocks not just robots but also unregistered visitors. If that is your intention, fine. If you want to allow everyone to see a particular forum and just don't want that forum's threads appearing in search engines then I don't believe you can do that with vB.

[nhawk]

Originally Posted by Macsee

Note that the blocking suggested above blocks not just robots but also unregistered visitors. If that is your intention, fine. If you want to allow everyone to see a particular forum and just don't want that forum's threads appearing in search engines then I don't believe you can do that with vB.

I said that.