Go Back   vb.org Archive > vBulletin 3 Discussion > vB3 General Discussions
  #1  
Old 01-14-2007, 07:25 PM
bo0oost bo0oost is offline
 
Join Date: Oct 2006
Posts: 60
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default My Robots.txt... what does yours look like?

Here is my robots.txt... my site is SEO'd, so I blocked the archive since I don't want the bots to see duplicate content.

What does your robots.txt look like?

User-agent: *
Disallow: /forums/archive/index.php
Disallow: /forums/register.php
Disallow: /forums/newthread.php
Disallow: /forums/sendmessage.php
Disallow: /forums/online.php
Disallow: /orderstatus/
Disallow: /forums/newreply.php
Disallow: /forums/usercp.php
Disallow: /forums/faq.php
Disallow: /forums/reputation.php
Disallow: /forums/report.php
Disallow: /forums/printthread.php
Disallow: /forums/sendmessage.php
Disallow: /forums/subscription.php
Disallow: /forums/search.php
Disallow: /forums/profile.php
Disallow: /forums/private.php
Disallow: /photos/uploadphoto.php
Disallow: /classifieds/uploadproduct.php

no one has any robots.txt files?
Reply With Quote
  #2  
Old 12-05-2008, 09:59 AM
bongwater's Avatar
bongwater bongwater is offline
 
Join Date: Nov 2005
Location: south Florida
Posts: 99
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

User-agent: Mediapartners-Google*
Disallow:

User-agent: Fasterfox
Disallow: /

User-agent: *
Disallow: /cgi-bin/
Disallow: /forums/admincp/
Disallow: /forums/clientscript/
Disallow: /forums/cpstyles/
Disallow: /forums/customavatars/
Disallow: /forums/customprofilepics/
Disallow: /forums/images/
Disallow: /forums/modcp/
Disallow: /forums/ajax.php
Disallow: /forums/attachment.php
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/global.php
Disallow: /forums/image.php
Disallow: /forums/inlinemod.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newattachment.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/private.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/sendmessage.php
Disallow: /forums/showgroups.php
Disallow: /forums/subscription.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/usernote.php
Reply With Quote
  #3  
Old 12-05-2008, 12:32 PM
soundbarrierpro soundbarrierpro is offline
 
Join Date: Nov 2007
Posts: 767
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Mine has nothing on it. Should I be disallowing for SEO? What is the benefit?
Reply With Quote
  #4  
Old 12-05-2008, 02:51 PM
Zachariah's Avatar
Zachariah Zachariah is offline
 
Join Date: Feb 2002
Location: Canoga Park, CA
Posts: 2,125
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally Posted by soundbarrierpro View Post
Mine has nothing on it. Should I be disallowing for SEO? What is the benefit?
Most follow "robots.txt" to keep them out of areas that do not have data.

EX:
Disallow: /forums/admincp/

This tells a spider that that "Admin" sub folder is something it should not bother indexing.
- You do not have data there for a guest to view, so tell spiders not to follow links or access.

When you create a map of files and folders the spiders should not bother with it speeds up your listing/indexing of data on your site you want the public to see vs. the spider following every link and hitting "no access" pages wasting bandwidth and time.
Reply With Quote
  #5  
Old 12-05-2008, 04:23 PM
soundbarrierpro soundbarrierpro is offline
 
Join Date: Nov 2007
Posts: 767
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally Posted by Zachariah View Post
Most follow "robots.txt" to keep them out of areas that do not have data.

EX:
Disallow: /forums/admincp/

This tells a spider that that "Admin" sub folder is something it should not bother indexing.
- You do not have data there for a guest to view, so tell spiders not to follow links or access.

When you create a map of files and folders the spiders should not bother with it speeds up your listing/indexing of data on your site you want the public to see vs. the spider following every link and hitting "no access" pages wasting bandwidth and time.
Very nicely received. Thank you for your answer. I've never understood the robots.txt but now do.:up:
Reply With Quote
  #6  
Old 12-05-2008, 04:30 PM
Lynne's Avatar
Lynne Lynne is offline
 
Join Date: Sep 2004
Location: California/Idaho
Posts: 41,180
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

There should not be any links to follow to your modcp or admincp so you shouldn't need to put them in your robots.txt Also, you should be changing the name of your modcp and admincp and putting the new name in there just allows anyone else to download your robots.txt file and see what the new name is.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 09:35 PM.


Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.
X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.03464 seconds
  • Memory Usage 2,198KB
  • Queries Executed 13 (?)
More Information
Template Usage:
  • (1)SHOWTHREAD
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)ad_showthread_beforeqr
  • (1)ad_showthread_firstpost
  • (1)ad_showthread_firstpost_sig
  • (1)ad_showthread_firstpost_start
  • (2)bbcode_quote
  • (1)footer
  • (1)forumjump
  • (1)forumrules
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (1)navbar
  • (3)navbar_link
  • (120)option
  • (6)post_thanks_box
  • (6)post_thanks_button
  • (1)post_thanks_javascript
  • (1)post_thanks_navbar_search
  • (6)post_thanks_postbit_info
  • (6)postbit
  • (6)postbit_onlinestatus
  • (6)postbit_wrapper
  • (1)spacer_close
  • (1)spacer_open
  • (1)tagbit_wrapper 

Phrase Groups Available:
  • global
  • inlinemod
  • postbit
  • posting
  • reputationlevel
  • showthread
Included Files:
  • ./showthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/functions_bigthree.php
  • ./includes/class_postbit.php
  • ./includes/class_bbcode.php
  • ./includes/functions_reputation.php
  • ./includes/functions_post_thanks.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_postinfo_query
  • fetch_postinfo
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • showthread_start
  • showthread_getinfo
  • forumjump
  • showthread_post_start
  • showthread_query_postids
  • showthread_query
  • bbcode_fetch_tags
  • bbcode_create
  • showthread_postbit_create
  • postbit_factory
  • postbit_display_start
  • post_thanks_function_post_thanks_off_start
  • post_thanks_function_post_thanks_off_end
  • post_thanks_function_fetch_thanks_start
  • post_thanks_function_fetch_thanks_end
  • post_thanks_function_thanked_already_start
  • post_thanks_function_thanked_already_end
  • fetch_musername
  • postbit_imicons
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • postbit_display_complete
  • post_thanks_function_can_thank_this_post_start
  • tag_fetchbit_complete
  • forumrules
  • navbits
  • navbits_complete
  • showthread_complete