My Robots.txt... what does yours look like?

[bo0oost]

Here is my robots.txt... my site is SEO'd, so I blocked the archive since I don't want the bots to see duplicate content.

What does your robots.txt look like?

User-agent: *
Disallow: /forums/archive/index.php
Disallow: /forums/register.php
Disallow: /forums/newthread.php
Disallow: /forums/sendmessage.php
Disallow: /forums/online.php
Disallow: /orderstatus/
Disallow: /forums/newreply.php
Disallow: /forums/usercp.php
Disallow: /forums/faq.php
Disallow: /forums/reputation.php
Disallow: /forums/report.php
Disallow: /forums/printthread.php
Disallow: /forums/sendmessage.php
Disallow: /forums/subscription.php
Disallow: /forums/search.php
Disallow: /forums/profile.php
Disallow: /forums/private.php
Disallow: /photos/uploadphoto.php
Disallow: /classifieds/uploadproduct.php

no one has any robots.txt files?

[bongwater]

User-agent: Mediapartners-Google*
Disallow:

User-agent: Fasterfox
Disallow: /

User-agent: *
Disallow: /cgi-bin/
Disallow: /forums/admincp/
Disallow: /forums/clientscript/
Disallow: /forums/cpstyles/
Disallow: /forums/customavatars/
Disallow: /forums/customprofilepics/
Disallow: /forums/images/
Disallow: /forums/modcp/
Disallow: /forums/ajax.php
Disallow: /forums/attachment.php
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/global.php
Disallow: /forums/image.php
Disallow: /forums/inlinemod.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newattachment.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/private.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/sendmessage.php
Disallow: /forums/showgroups.php
Disallow: /forums/subscription.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/usernote.php

[soundbarrierpro]

Mine has nothing on it. Should I be disallowing for SEO? What is the benefit?

[Zachariah]

Originally Posted by soundbarrierpro

Mine has nothing on it. Should I be disallowing for SEO? What is the benefit?

Most follow "robots.txt" to keep them out of areas that do not have data.

EX:
Disallow: /forums/admincp/

This tells a spider that that "Admin" sub folder is something it should not bother indexing.
- You do not have data there for a guest to view, so tell spiders not to follow links or access.

When you create a map of files and folders the spiders should not bother with it speeds up your listing/indexing of data on your site you want the public to see vs. the spider following every link and hitting "no access" pages wasting bandwidth and time.

[soundbarrierpro]

Originally Posted by Zachariah

Most follow "robots.txt" to keep them out of areas that do not have data.

EX:
Disallow: /forums/admincp/

This tells a spider that that "Admin" sub folder is something it should not bother indexing.
- You do not have data there for a guest to view, so tell spiders not to follow links or access.

When you create a map of files and folders the spiders should not bother with it speeds up your listing/indexing of data on your site you want the public to see vs. the spider following every link and hitting "no access" pages wasting bandwidth and time.

Very nicely received. Thank you for your answer. I've never understood the robots.txt but now do.:up:

[Lynne]

There should not be any links to follow to your modcp or admincp so you shouldn't need to put them in your robots.txt Also, you should be changing the name of your modcp and admincp and putting the new name in there just allows anyone else to download your robots.txt file and see what the new name is.