Go Back   vb.org Archive > vBulletin 3 Discussion > vB3 General Discussions
FAQ Community Calendar Today's Posts Search

Reply
 
Thread Tools Display Modes
  #1  
Old 03-16-2008, 07:23 PM
logicuk logicuk is offline
 
Join Date: Jan 2008
Posts: 327
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default perfect robots.txt for vb 3.7?

Does anyone care to share the perfect robots.txt for vb 3.7

here is what i have

Code:
User-agent: *
Disallow: /admincp/
Disallow: /archive/
Disallow: /backup/
Disallow: /announcement.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /faq.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /misc.php
Disallow: /modcp/
Disallow: /moderator.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /subscription.php
Disallow: /subscriptions.php
Disallow: /threadrate.php
Disallow: /usercp.php
Reply With Quote
  #2  
Old 03-16-2008, 07:54 PM
Boofo's Avatar
Boofo Boofo is offline
 
Join Date: Mar 2002
Location: Des Moines, IA (USA)
Posts: 15,776
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

You should have that in the domain root.
Reply With Quote
  #3  
Old 03-17-2008, 03:25 AM
Bradley_Wint Bradley_Wint is offline
 
Join Date: Jul 2007
Posts: 543
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

don't mind me sounding like a newbie here but i added the content to a robots.txt file, upped it to my server html root folder like 8 hrs ago..now when i check my who's online list, i still see google bots murdering my calender.php, login.php...etc...

anything i did wrong?
Reply With Quote
  #4  
Old 03-17-2008, 03:38 AM
blike2 blike2 is offline
 
Join Date: Jul 2002
Posts: 8
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Google only checks robots.txt every day or two, I think. Give it a little time.
Reply With Quote
  #5  
Old 03-17-2008, 03:39 AM
Bradley_Wint Bradley_Wint is offline
 
Join Date: Jul 2007
Posts: 543
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally Posted by blike2 View Post
Google only checks robots.txt every day or two, I think. Give it a little time.
thanks mate
Reply With Quote
  #6  
Old 03-17-2008, 04:42 AM
SEOvB's Avatar
SEOvB SEOvB is offline
 
Join Date: May 2007
Location: Indianapolis
Posts: 2,451
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

once every 24hrs
Reply With Quote
  #7  
Old 03-17-2008, 04:45 AM
Bradley_Wint Bradley_Wint is offline
 
Join Date: Jul 2007
Posts: 543
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

yes apparently it worked like 5 minutes after i posted... only the important files like forumdisplay, showpost, showthread etc we being scanned. thanks for this mate. cheers.
Reply With Quote
  #8  
Old 03-17-2008, 05:28 AM
jambo_1969 jambo_1969 is offline
 
Join Date: May 2007
Posts: 224
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Make sure you add in

User-agent: Fasterfox
Disallow: /

I managed to catch a thread on my board where people were discussing have Fasterfox set as turbo charged...my loads went through the roof !!
Reply With Quote
  #9  
Old 03-17-2008, 05:31 AM
Boofo's Avatar
Boofo Boofo is offline
 
Join Date: Mar 2002
Location: Des Moines, IA (USA)
Posts: 15,776
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Can we combine these 2?

Quote:
User-Agent: Googlebot-Image
Disallow: /

User-agent: Fasterfox
Disallow: /
Like this?

Quote:
User-Agent: Googlebot-Image
User-agent: Fasterfox
Disallow: /
Reply With Quote
  #10  
Old 03-17-2008, 05:36 AM
jambo_1969 jambo_1969 is offline
 
Join Date: May 2007
Posts: 224
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally Posted by Boofo View Post
Can we combine these 2?



Like this?
Not sure...

This is how mine looks (part of it anyway)
Code:
User-agent: Fasterfox
Disallow: /
User-agent: * 
Disallow: /.k.k.k./
?
Reply With Quote
Reply


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 08:56 AM.


Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.
X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.06704 seconds
  • Memory Usage 2,248KB
  • Queries Executed 13 (?)
More Information
Template Usage:
  • (1)SHOWTHREAD
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)ad_showthread_beforeqr
  • (1)ad_showthread_firstpost
  • (1)ad_showthread_firstpost_sig
  • (1)ad_showthread_firstpost_start
  • (2)bbcode_code
  • (4)bbcode_quote
  • (1)footer
  • (1)forumjump
  • (1)forumrules
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (1)navbar
  • (3)navbar_link
  • (120)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (2)pagenav_pagelink
  • (10)post_thanks_box
  • (10)post_thanks_button
  • (1)post_thanks_javascript
  • (1)post_thanks_navbar_search
  • (10)post_thanks_postbit_info
  • (10)postbit
  • (10)postbit_onlinestatus
  • (10)postbit_wrapper
  • (1)spacer_close
  • (1)spacer_open
  • (1)tagbit_wrapper 

Phrase Groups Available:
  • global
  • inlinemod
  • postbit
  • posting
  • reputationlevel
  • showthread
Included Files:
  • ./showthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/functions_bigthree.php
  • ./includes/class_postbit.php
  • ./includes/class_bbcode.php
  • ./includes/functions_reputation.php
  • ./includes/functions_post_thanks.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_postinfo_query
  • fetch_postinfo
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • showthread_start
  • showthread_getinfo
  • forumjump
  • showthread_post_start
  • showthread_query_postids
  • showthread_query
  • bbcode_fetch_tags
  • bbcode_create
  • showthread_postbit_create
  • postbit_factory
  • postbit_display_start
  • post_thanks_function_post_thanks_off_start
  • post_thanks_function_post_thanks_off_end
  • post_thanks_function_fetch_thanks_start
  • post_thanks_function_fetch_thanks_end
  • post_thanks_function_thanked_already_start
  • post_thanks_function_thanked_already_end
  • fetch_musername
  • postbit_imicons
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • postbit_display_complete
  • post_thanks_function_can_thank_this_post_start
  • pagenav_page
  • pagenav_complete
  • tag_fetchbit_complete
  • forumrules
  • navbits
  • navbits_complete
  • showthread_complete