Go Back   vb.org Archive > vBulletin 4 Discussion > vB4 General Discussions

Reply
 
Thread Tools Display Modes
  #11  
Old 10-06-2013, 02:11 PM
DemOnstar's Avatar
DemOnstar DemOnstar is offline
 
Join Date: Dec 2012
Posts: 859
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Ok, from the example above, I am currently using this.

Code:
Sitemap: https://www.mysite.com/sitemap_index.xml.gz


User-agent: Mediapartners-Google
Disallow:


User-agent: Baiduspider
Disallow: /



User-agent: *
Disallow: clientscript/
Disallow: cpstyles/
Disallow: customavatars/
Disallow: customgroupicons/
Disallow: customprofilepics/
Disallow: customsignaturepics/
Disallow: forumrunner/
Disallow: images/
Disallow: includes/
Disallow: members/
Disallow: sitemap/
Disallow: ajax.php
Disallow: attachment.php
Disallow: calendar.php
Disallow: cron.php
Disallow: editpost.php
Disallow: global.php
Disallow: image.php
Disallow: inlinemod.php
Disallow: joinrequests.php
Disallow: login.php
Disallow: member.php
Disallow: memberlist.php
Disallow: misc.php
Disallow: moderator.php
Disallow: newattachment.php
Disallow: newreply.php
Disallow: newthread.php
Disallow: online.php
Disallow: poll.php
Disallow: postings.php
Disallow: printthread.php
Disallow: private.php
Disallow: profile.php
Disallow: register.php
Disallow: report.php
Disallow: reputation.php
Disallow: search.php
Disallow: sendmessage.php
Disallow: subscription.php
Disallow: threadrate.php
Disallow: thumbs.php
Disallow: thumb_center.php
Disallow: usercp.php
Disallow: usernote.php
Disallow: visitormessage.php
Does that seem in order?

I have removed some stuff..and added other things..

Thanks...
Reply With Quote
  #12  
Old 11-10-2013, 06:14 PM
gsmlover4u's Avatar
gsmlover4u gsmlover4u is offline
 
Join Date: Jan 2007
Posts: 348
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

what is benifit to disallow every thing
i think then search engine will not crawl website ?
Reply With Quote
  #13  
Old 11-10-2013, 06:23 PM
Max Taxable's Avatar
Max Taxable Max Taxable is offline
 
Join Date: Feb 2011
Posts: 3,134
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

There is no reason to have modcp or admincp in robots.txt - first of all this file is public, so you are telling the script kiddies out there where these pages are, no matter what you have named them.

Second and most important, bad bots ignore robots.txt anyway. Yes that's right - robots.txt isn't a law, or a automated restriction, it's a "please don't index this" page with no teeth at all.

Robots.txt is alot like gun laws - only the law abiding obey them or even pay any attention to them.

If google and other good bots index these pages, they generally won't show in search results because they have no content relevance.

If you want the bad bots out of your life installing THIS GREAT MOD is the way to go.
Reply With Quote
Благодарность от:
gsmlover4u
  #14  
Old 03-07-2016, 10:27 AM
webmastersun's Avatar
webmastersun webmastersun is offline
 
Join Date: Oct 2013
Location: www.webmastersun.com
Posts: 433
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally Posted by pityocamptes View Post
Does VB4 or this site provide an up-to-date robot.txt file? Any help appreciated.
You should do robots.txt depend on your forum, which sections/parts you want to block or allow robots then add
Disallow or Allow in your robots.txt file

Since vBulletin community is a forum to discuss all things about vBulletin, not SEO then I would suggest some SEO experts on this SEO forum
http://www.webmastersun.com/forums/1...e-Optimization

They can support solve more SEO problems and give you best solutions for your SEO questions.
Quote:
Originally Posted by blake247 View Post
Disallow: /images/
Disallow: /register.php
I am wondering why you want to block this folder and file? it's still useful for search engine crawl and index your images file and register page.

Hope it helps!
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 02:44 PM.


Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2024, vBulletin Solutions Inc.
X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.03718 seconds
  • Memory Usage 2,194KB
  • Queries Executed 13 (?)
More Information
Template Usage:
  • (1)SHOWTHREAD
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)ad_showthread_beforeqr
  • (1)ad_showthread_firstpost
  • (1)ad_showthread_firstpost_sig
  • (1)ad_showthread_firstpost_start
  • (1)bbcode_code
  • (2)bbcode_quote
  • (1)footer
  • (1)forumjump
  • (1)forumrules
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (1)navbar
  • (3)navbar_link
  • (120)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (1)pagenav_pagelink
  • (4)post_thanks_box
  • (1)post_thanks_box_bit
  • (4)post_thanks_button
  • (1)post_thanks_javascript
  • (1)post_thanks_navbar_search
  • (1)post_thanks_postbit
  • (4)post_thanks_postbit_info
  • (4)postbit
  • (4)postbit_onlinestatus
  • (4)postbit_wrapper
  • (1)spacer_close
  • (1)spacer_open
  • (1)tagbit_wrapper 

Phrase Groups Available:
  • global
  • inlinemod
  • postbit
  • posting
  • reputationlevel
  • showthread
Included Files:
  • ./showthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/functions_bigthree.php
  • ./includes/class_postbit.php
  • ./includes/class_bbcode.php
  • ./includes/functions_reputation.php
  • ./includes/functions_post_thanks.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_postinfo_query
  • fetch_postinfo
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • showthread_start
  • showthread_getinfo
  • forumjump
  • showthread_post_start
  • showthread_query_postids
  • showthread_query
  • bbcode_fetch_tags
  • bbcode_create
  • showthread_postbit_create
  • postbit_factory
  • postbit_display_start
  • post_thanks_function_post_thanks_off_start
  • post_thanks_function_post_thanks_off_end
  • post_thanks_function_fetch_thanks_start
  • fetch_musername
  • post_thanks_function_fetch_thanks_end
  • post_thanks_function_thanked_already_start
  • post_thanks_function_thanked_already_end
  • postbit_imicons
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • postbit_display_complete
  • post_thanks_function_can_thank_this_post_start
  • post_thanks_function_fetch_thanks_bit_start
  • post_thanks_function_show_thanks_date_start
  • post_thanks_function_show_thanks_date_end
  • post_thanks_function_fetch_thanks_bit_end
  • post_thanks_function_fetch_post_thanks_template_start
  • post_thanks_function_fetch_post_thanks_template_end
  • pagenav_page
  • pagenav_complete
  • tag_fetchbit_complete
  • forumrules
  • navbits
  • navbits_complete
  • showthread_complete