Yet another ROBOTS.txt file question (sorry)

[Macsee]

You said what, that it can't be done with vB?

In fact, I made that post hoping someone would tell me I was wrong and point to a way one could allow visitors into a forum but prevent that forum's threads from showing up in the SEs.

[ForceHSS]

If the forum is set to private spiders will not be able to access it anyway

[Macsee]

Setting it to private blocks unregistered visitors.

I know the logic may sound screwed up, but there is sense in such a setup. For example, where you want to allow links in the sub-forum but want to dissuade people posting links in there for the SEO benefit.

[nhawk]

Originally Posted by Macsee

You said what, that it can't be done with vB?

In fact, I made that post hoping someone would tell me I was wrong and point to a way one could allow visitors into a forum but prevent that forum's threads from showing up in the SEs.

I said the this part of your first post...

Originally Posted by Macsee

.....

Note that the blocking suggested above blocks not just robots but also unregistered visitors. If that is your intention, fine....

[invitezone]

ok all good info thanks for your time everyone.
I actually want a small section of my forum to be PUBLIC and I want search engines to index it.
I just dont want to have 50 bots killing my bandwidth.

any answer to that?

Thanks a million for your help.

[Nichtofen]

bump...

1. Create a robots.txt to eliminate pages like profiles, search, etc from spiders (This will also help eliminate profiles and other pages from coming up in searches by potential traffic.)

2. Create permissions per forum for unregistered visitors like 'nhawk' described and be careful as visitors will not be able to view these forums either depending on configurations.

3. If you really wish to control which robots/spiders are accessing your forums and utilizing bandwidth, consider this add on. I do however recommend you read all 350 posts in order to guarantee your success. There is a lot to read on user agents even beyond the resources available within the tread in order to properly use this program and ensure you are not shutting out spiders that could help your traffic and cause.


Ban Spiders by User Agent by Simon Lloyd

[zascok]

here is the start for you, then get robots you don't like covered by
User-agent: Name of Robot
Disallow: /

or ban them ^ just like said above

robot.txt

User-agent: *
Crawl-delay: 10
Disallow: /*.js
Disallow: /clientscript/
Disallow: /customgroupicons/
Disallow: /packages/
Disallow: /signaturepics/
Disallow: /customprofilepics/
Disallow: /store_sitemap/
Disallow: /vb/
Disallow: /cpstyles/
Disallow: /cron.php
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /includes/
Disallow: /images/
Disallow: /ajax.php
Disallow: /album.php
Disallow: /announcement.php
Disallow: /api.php
Disallow: /apichain.php
Disallow: /asset.php
Disallow: /assetmanage.php
Disallow: /attachment.php
Disallow: /attachment_inlinemod.php
Disallow: /blog_attachment.php
Disallow: /calendar.php
Disallow: /ckeditor.php
Disallow: /clear.gif
Disallow: /converse.php
Disallow: /cron.php
Disallow: /css.php
Disallow: /editor.php
Disallow: /editpost.php
Disallow: /entry.php
Disallow: /external.php
Disallow: /faq.php
Disallow: /favicon.ico
Disallow: /global.php
Disallow: /group.php
Disallow: /group_inlinemod.php
Disallow: /groupsubscription.php
Disallow: /image.php
Disallow: /infraction.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /LICENSE
Disallow: /list.php
Disallow: /login.php
Disallow: /member.php
Disallow: /member_inlinemod.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /mobile.php
Disallow: /moderation.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /payment_gateway.php
Disallow: /payments.php
Disallow: /picture.php
Disallow: /picture_inlinemod.php
Disallow: /picturecomment.php
Disallow: /poll.php
Disallow: /posthistory.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /threadtag.php
Disallow: /uploadprogress.gif
Disallow: /usercp.php
Disallow: /usernote.php
Disallow: /visitormessage.php
Disallow: /widget.php
Disallow: /xmlsitemap.php

[invitezone]

thanks a lot for this zascok.
if I dont want to ban any spiders can I just copy and paste this into a txt file and leave it at that?

[zascok]

yup and nope you can't leave it at that you gotta up it into the root of your forum

[Nichtofen]

Indeed. The file should be located at www.yourdomain.com/robot.txt . Right in your root assuming that you have that access. That is where Google, as well as other well behaved bots will look for it. It gets more complicated if you are on a shared server through a provider that gives you a default address such as www.sharedservercompany.com/yourusername or something similar. In that situation, you would require you host to assist if they are able.

--------------- Added [DATE]1344202148[/DATE] at [TIME]1344202148[/TIME] ---------------

Curiously,
Within your code you do not have a "forums/" prefix on your items. That would be required if your forum was located within a forums folder in the root, correct?