vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   Forum and Server Management (https://vborg.vbsupport.ru/forumdisplay.php?f=232)
-   -   Question: Robots.txt File (https://vborg.vbsupport.ru/showthread.php?t=302782)

Narutoboy 09-29-2013 03:02 AM
Question: Robots.txt File
 
Is there anything that I've disallowed that will hurt me as far my rankings in Google?


Quote:
User-agent: *
Disallow: /admincp/
Disallow: /arcade/
Disallow: /archive/
Disallow: /cgi-bin/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /customgroupicons/
Disallow: /customprofilepics/
Disallow: /images/
Disallow: /includes/
Disallow: /modcp/
Disallow: /packages/
Disallow: /signaturepics/
Disallow: /vb/
Disallow: /ajax.php
Disallow: /album.php
Disallow: /announcement.php
Disallow: /asset.php
Disallow: /assetmanage.php
Disallow: /attachment.php
Disallow: /attachment_inlinemod.php
Disallow: /blog_ajax.php
Disallow: /blog_external.php
Disallow: /blog_inlinemod.php
Disallow: /blog_report.php
Disallow: /blog_search.php
Disallow: /blog_subscription.php
Disallow: /blog_usercp.php
Disallow: /calendar.php
Disallow: /clear.gif
Disallow: /converse.php
Disallow: /cron.php
Disallow: /editor.php
Disallow: /editpost.php
Disallow: /external.php
Disallow: /faq.php
Disallow: /global.php
Disallow: /group_inlinemod.php
Disallow: /groupsubscription.php
Disallow: /image.php
Disallow: /infraction.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /list.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderation.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /online.php
Disallow: /payment_gateway.php
Disallow: /payments.php
Disallow: /picture.php
Disallow: /picture_inlinemod.php
Disallow: /picturecomment.php
Disallow: /posthistory.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /sendtofriend.php
Disallow: /showgroups.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php
Disallow: /visitormessage.php
Disallow: /widget.php

CAG CheechDogg 09-29-2013 03:59 AM
Leave your admincp, modcp out of the robots.txt file, those are areas you don't even want the bots to know where they are ...

If anything put a htacess password on them

Narutoboy 09-29-2013 03:05 PM
Quote:
Originally Posted by CAG CheechDogg (Post 2448732)
Leave your admincp, modcp out of the robots.txt file, those are areas you don't even want the bots to know where they are ...

If anything put a htacess password on them
Thanks, didn't think about that.

socialteenz 09-29-2013 05:25 PM
Hello, feel free to include your sitemap at the end of the robots.txt.

Also, it is better to add something like this at the top, since we know google bot's name.

Code:
User-agent: Mediapartners-Google
Disallow:

User-agent: *
Crawl-delay:  30
Add your list here.

Sitemap: http://sitemapurl
You can validate your sitemap here.


All times are GMT. The time now is 03:36 PM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01003 seconds
  • Memory Usage 1,724KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)bbcode_code_printable
  • (2)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (4)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 
Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 
Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete