vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vB4 General Discussions (https://vborg.vbsupport.ru/forumdisplay.php?f=251)
-   -   Updated robot.txt file? (https://vborg.vbsupport.ru/showthread.php?t=302483)

DemOnstar 10-06-2013 02:11 PM

Ok, from the example above, I am currently using this.

Code:

Sitemap: https://www.mysite.com/sitemap_index.xml.gz


User-agent: Mediapartners-Google
Disallow:


User-agent: Baiduspider
Disallow: /



User-agent: *
Disallow: clientscript/
Disallow: cpstyles/
Disallow: customavatars/
Disallow: customgroupicons/
Disallow: customprofilepics/
Disallow: customsignaturepics/
Disallow: forumrunner/
Disallow: images/
Disallow: includes/
Disallow: members/
Disallow: sitemap/
Disallow: ajax.php
Disallow: attachment.php
Disallow: calendar.php
Disallow: cron.php
Disallow: editpost.php
Disallow: global.php
Disallow: image.php
Disallow: inlinemod.php
Disallow: joinrequests.php
Disallow: login.php
Disallow: member.php
Disallow: memberlist.php
Disallow: misc.php
Disallow: moderator.php
Disallow: newattachment.php
Disallow: newreply.php
Disallow: newthread.php
Disallow: online.php
Disallow: poll.php
Disallow: postings.php
Disallow: printthread.php
Disallow: private.php
Disallow: profile.php
Disallow: register.php
Disallow: report.php
Disallow: reputation.php
Disallow: search.php
Disallow: sendmessage.php
Disallow: subscription.php
Disallow: threadrate.php
Disallow: thumbs.php
Disallow: thumb_center.php
Disallow: usercp.php
Disallow: usernote.php
Disallow: visitormessage.php

Does that seem in order?

I have removed some stuff..and added other things..

Thanks...

gsmlover4u 11-10-2013 06:14 PM

what is benifit to disallow every thing
i think then search engine will not crawl website ?

Max Taxable 11-10-2013 06:23 PM

There is no reason to have modcp or admincp in robots.txt - first of all this file is public, so you are telling the script kiddies out there where these pages are, no matter what you have named them.

Second and most important, bad bots ignore robots.txt anyway. Yes that's right - robots.txt isn't a law, or a automated restriction, it's a "please don't index this" page with no teeth at all.

Robots.txt is alot like gun laws - only the law abiding obey them or even pay any attention to them.

If google and other good bots index these pages, they generally won't show in search results because they have no content relevance.

If you want the bad bots out of your life installing THIS GREAT MOD is the way to go.

webmastersun 03-07-2016 10:27 AM

Quote:

Originally Posted by pityocamptes (Post 2446909)
Does VB4 or this site provide an up-to-date robot.txt file? Any help appreciated.

You should do robots.txt depend on your forum, which sections/parts you want to block or allow robots then add
Disallow or Allow in your robots.txt file

Since vBulletin community is a forum to discuss all things about vBulletin, not SEO then I would suggest some SEO experts on this SEO forum
http://www.webmastersun.com/forums/1...e-Optimization

They can support solve more SEO problems and give you best solutions for your SEO questions.
Quote:

Originally Posted by blake247 (Post 2447224)
Disallow: /images/
Disallow: /register.php

I am wondering why you want to block this folder and file? it's still useful for search engine crawl and index your images file and register page.

Hope it helps!


All times are GMT. The time now is 01:59 PM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01022 seconds
  • Memory Usage 1,727KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)bbcode_code_printable
  • (2)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (1)pagenav_pagelink
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (4)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 

Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • pagenav_page
  • pagenav_complete
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete