vb.org Archive
Page 2 of 2 1 2
Show 40 post(s) from this thread on one page

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vB3 General Discussions (https://vborg.vbsupport.ru/forumdisplay.php?f=111)
-   -   549 spiders on my site (https://vborg.vbsupport.ru/showthread.php?t=202367)

dtv100 02-09-2009 05:15 AM
here a copy of my robot:

Quote:
User-agent: *
User-agent: Slurp
User-agent: Mediapartners-Google*

Disallow: /forums/referrers.php
Disallow: /forums/arcade.php
Disallow: /forums/ajax_cron.php
Disallow: /forums/ajax.php
Disallow: /forums/attachment.php
Disallow: /forums/calendar.php
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/global.php
Disallow: /forums/image.php
Disallow: /forums/inlinemod.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/member.php
Disallow: /forums/memberlist.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newattachment.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/post_thanks.php
Disallow: /forums/payment_gateway.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/rules.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/showgroups.php
Disallow: /forums/subscription.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/admin/
Disallow: /forums/infernoshout.php
Disallow: /forums/redbar
Disallow: /forums/chat
Disallow: /forums/members/
Disallow: /forums/member-*

Sitemap:http://www.domain.com/forums/sitemap_index.xml.gz

kNeeLy 02-09-2009 06:22 PM
so..Is it better to disallow a few of your links?

Brandon Sheley 02-09-2009 06:41 PM
Quote:
Originally Posted by kNeeLy (Post 1738942)
so..Is it better to disallow a few of your links?
You don't need the spiders checking out the pages or content that isn't directly related to your site ;)

The robots.txt file posted above is good.. Everyone should have a copy of that in their root domain.. Yousite.com/robots.txt

Replace the domain info for your own with regards to the sitemap ;)

here is the one in my root
http://www.vbulletinsetup.com/robots.txt
and since my forum is a subdomain, I have one for it as well
http://forum.vbulletinsetup.com/robots.txt

ilrglen 08-17-2009 01:32 PM
This may be a stupid quest but what is the gz for in these links to your sitemap?

Code:
# BEGIN XML-SITEMAP-PLUGIN
Sitemap: http://www.vbulletinsetup.com/sitemap.xml.gz
Sitemap: http://forum.vbulletinsetup.com/sitemap_index.xml.gz
# END XML-SITEMAP-PLUGIN

User-agent: Mediapartners-Google*
Disallow:
User-agent: *

Disallow: /pics
Disallow: /wp-admin
Disallow: /wp-login
Disallow: /wp-register
Disallow: /2006/
Disallow: /2007/
Disallow: /2008/


All times are GMT. The time now is 10:34 AM.
Page 2 of 2 1 2
Show 40 post(s) from this thread on one page

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.00984 seconds
  • Memory Usage 1,727KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)bbcode_code_printable
  • (2)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (1)pagenav_pagelink
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (4)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 
Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 
Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • pagenav_page
  • pagenav_complete
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete