vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vB3 General Discussions (https://vborg.vbsupport.ru/forumdisplay.php?f=111)
-   -   robots question (https://vborg.vbsupport.ru/showthread.php?t=216941)

Stormraven 06-23-2009 07:26 PM
robots question
 
I'm trying to optimize my vbulletin for seo with robots.txt. But I just have one small question. I saw some robots.txt file contents that Redlinemotorsports posted in another thread, this is the contents of my robots.txt file:

Code:
User-agent: *
Disallow: /archive/
Disallow: /lo-fi/
Disallow: /general-announcements/
Disallow: /cgi-bin/
Disallow: /clientscript/
Disallow: /includes/
Disallow: /install/
Disallow: /sitemap/
Disallow: /tags/
Disallow: /ads/
Disallow: /video/
Disallow: /games/
Disallow: /partners/
Disallow: /printthread.php
Disallow: /subscription.php
Disallow: /payments.php
Disallow: /profile.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /online.php
Disallow: /faq.php
Disallow: /calendar.php
Disallow: /private.php
Disallow: /sendmessage.php
Disallow: /sendmessage.php?do=
Disallow: /showgroups.php
Disallow: /reputation.php
Disallow: /report.php
Disallow: /threadrate.php
Disallow: /postings.php
Disallow: /newthread.php
Disallow: /search.php
Disallow: /newreply.php
Disallow: /register.php
Disallow: /login.php
Disallow: /faq.php
Disallow: /image.php
Disallow: /cron.php
Disallow: /joinrequests.php
Disallow: /usercp.php
Disallow: /vbseocp.php
but this is what Redlinemotorsports posted:

Code:
User-agent: *
Disallow: /forums/ajax.php
Disallow: /forums/attachment.php
Disallow: /forums/calendar.php
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/global.php
Disallow: /forums/image.php
Disallow: /forums/inlinemod.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/member.php
Disallow: /forums/memberlist.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newattachment.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/showgroups.php
Disallow: /forums/subscription.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/usernote.php

But my question is, what is the difference between the two? do they both do the same thing? im trying to make google focus on main fresh content but am i duplicating things?

HMBeaty 06-24-2009 02:55 AM
Answered via Instant Messenger.

If anyone else needs the answer to this, its yes. Basically both that are listed, will do the same thing, but the 2nd one (the one I posted) is more for people who have their forums installed to yoursite.com/forums/. The FIRST one, is for people who have their forums installed to their root directory (yoursite.com/).

However, you'll see alot of people using the 2nd one (the one I posted) since its the more common one. Just paste that into your robots.txt file and remove what you want crawled, and add what you don't want crawled. It's that easy :D

BSMedia 06-24-2009 06:03 AM
Nice robots.txt, though it seems a little out of date and missing some files like tags.php albums.php and groups.php you can even take it a little further and block off your entire includes folder (I've seen vBulletin javascript get indexed :roll:) and the clientscript folders if you don't want your images cralwed, also block that off


Things to remember:
  • Robots.txt can only be served from your public_html (or www) folder. It can't not be placed in a sub-folder on your domain.
  • If your vBulletin isn't in /forums be sure to update the path to the correct location or else it won't do any good.

Stormraven 06-24-2009 03:59 PM
Which one do you mean is outdated MSMedia?

Kind Regards
Stormraven.

BSMedia 06-24-2009 05:31 PM
Both of them are.

There are simply missing 3 .php files thats all

Add

Disallow: tags.php
Disallow: groups.php
Disallow: albums.php
Disallow: clientscript/
Disallow: includes/

And that should pretty much fix you right up ;)


All times are GMT. The time now is 02:43 AM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.00964 seconds
  • Memory Usage 1,722KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (2)bbcode_code_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (5)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 
Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 
Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete