vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   Forum and Server Management (https://vborg.vbsupport.ru/forumdisplay.php?f=232)
-   -   robots.txt for 3.8.2 - Any Ideas? (https://vborg.vbsupport.ru/showthread.php?t=211965)

vbplusme 04-26-2009 11:45 AM

I can exclude it from the sitemap for sure but I thought it was pretty strange to be referencing archives from 1970 through current. The links looks like:

hxxp://www.mysite.com/blog.php?do=list&m=12&y=1970
hxxp://www.mysite.com/blog.php?u=1&m=12&y=1970
hxxp://www.mysite.com/blog.php?do=list&m=11&y=1970
hxxp://www.mysite.com/blog.php?u=1&m=11&y=1970
hxxp://www.mysite.com/blog.php?do=list&m=10&y=1970
hxxp://www.mysite.com/blog.php?u=1&m=10&y=1970
hxxp://www.mysite.com/blog.php?do=list&m=9&y=1970
hxxp://www.mysite.com/blog.php?u=1&m=9&y=1970
hxxp://www.mysite.com/blog.php?do=list&m=8&y=1970
hxxp://www.mysite.com/blog.php?u=1&m=8&y=1970
hxxp://www.mysite.com/blog.php?do=list&m=7&y=1970
hxxp://www.mysite.com/blog.php?u=1&m=7&y=1970
hxxp://www.mysite.com/blog.php?do=list&m=6&y=1970
hxxp://www.mysite.com/blog.php?u=1&m=6&y=1970
hxxp://www.mysite.com/blog.php?do=list&m=5&y=1970
hxxp://www.mysite.com/blog.php?u=1&m=5&y=1970
hxxp://www.mysite.com/blog.php?do=list&m=4&y=1970
hxxp://www.mysite.com/blog.php?u=1&m=4&y=1970
hxxp://www.mysite.com/blog.php?do=list&m=3&y=1970
hxxp://www.mysite.com/blog.php?u=1&m=3&y=1970
hxxp://www.mysite.com/blog.php?do=list&m=2&y=1970
hxxp://www.mysite.com/blog.php?u=1&m=2&y=1970
hxxp://www.mysite.com/blog.php?do=list&m=1&y=1970
hxxp://www.mysite.com/blog.php?u=1&m=1&y=1970

The sitemap software is finding this for every year to current?

BSMedia 04-26-2009 02:17 PM

Those are sorting URL's that google finds and goes crazy on. Block off blog.php and call it a day.

It will also do this for the calendar, usually on newer vBulletin sites for some reason if you use the site:http://sitename.com you'll see 500 pages of calender sorting URL's.

Its fine to have the warning messages in GWT that tell you access is restricted to those URL's, they provide no value what so ever to your sites rankings, and should be blocked off as with most sorting URL's since it will just be duplicated content from else where.

vbplusme 04-26-2009 02:47 PM

Thanks very much for the reply. I appreciate it. It fits with what I suspected on the duplicate content issues I am trying to solve. I will add blog.php to my robots.txt file and exclude it from my sitemap.

thanks again.

--------------- Added [DATE]1240771303[/DATE] at [TIME]1240771303[/TIME] ---------------

Thanks very much for the reply. I appreciate it. It fits with what I suspected on the duplicate content issues I am trying to solve. I will add blog.php to my robots.txt file and exclude it from my sitemap.

thanks again.

--------------- Added [DATE]1240797229[/DATE] at [TIME]1240797229[/TIME] ---------------

Quote:

Originally Posted by BSMedia (Post 1799309)
Those are sorting URL's that google finds and goes crazy on. Block off blog.php and call it a day.

It will also do this for the calendar, usually on newer vBulletin sites for some reason if you use the site:http://sitename.com you'll see 500 pages of calender sorting URL's.

Its fine to have the warning messages in GWT that tell you access is restricted to those URL's, they provide no value what so ever to your sites rankings, and should be blocked off as with most sorting URL's since it will just be duplicated content from else where.

Do Tags do the same thing, i.e. provide duplicate content with no value to the PR? I see a boat load of them in the sitemap too.

TIA for a reply.


All times are GMT. The time now is 03:16 PM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01080 seconds
  • Memory Usage 1,728KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (1)pagenav_pagelink
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (3)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 

Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • pagenav_page
  • pagenav_complete
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete