vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vB3 General Discussions (https://vborg.vbsupport.ru/forumdisplay.php?f=111)
-   -   Robots.txt file preventing search indexing? (https://vborg.vbsupport.ru/showthread.php?t=102507)

007 12-08-2005 04:49 PM
Robots.txt file preventing search indexing?
 
My sites no longer seem to be getting indexed. I'm fairly certain it has to do with my robots.txt file that I got from a thread here. The problem started shortly after I added this. Search engine bots never seem to get past the main page now. :-(

Code:
#ALL BOTS
User-agent: *
Disallow: /admincp/
Disallow: /ajax.php
Disallow: /arcade.php
Disallow: /attachments/
Disallow: /chat.php
Disallow: /chat/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /images/
Disallow: /includes/
Disallow: /install/
Disallow: /modcp/
Disallow: /subscriptions/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /announcement.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /external.php
Disallow: /faq.php
Disallow: /frm_attach
Disallow: /image.php
Disallow: /index.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php?
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /payment_gateway.php
Disallow: /payments.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /subscription.php
Disallow: /usercp.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php

Marco van Herwaarden 12-08-2005 05:42 PM
Disallow: /index.php

Remove that one.

007 12-08-2005 06:03 PM
The thread I got this from said that was ok to leave in.. I will remove it and see how it works. Hopefully that's the problem. :-)

007 12-13-2005 04:24 PM
I'm not sure if this helped. Is there anything else in there that would be stopping spiders? They don't seem to be getting past the index.

noppid 12-13-2005 04:36 PM
Quote:
Originally Posted by 007
I'm not sure if this helped. Is there anything else in there that would be stopping spiders? They don't seem to be getting past the index.

First off, that file assumes a ROOT install. Secondly, there are a bunch of files blocked that are beneficial to you in the SERPs in that list.

The fact that index.php would be there just absoluty blows me away! I'd throw it way and start over if I were you.

Here's a good base to start with. This file assumes vBulletin is in the folder named "forums".

Code:
User-agent: Mediapartners-Google*
Disallow:

User-agent: *
Disallow: /forums/admincp/
Disallow: /forums/modcp/
Disallow: /forums/install/
Disallow: /forums/clientscript/
Disallow: /forums/cpstyles/
Disallow: /forums/customavatars/
Disallow: /forums/customprofilepics/
Disallow: /forums/images/
Disallow: /forums/includes/
The first two lines assure that the adsense spider can go where ever it wants. The next lines exclude spiders from most of the places that the spiders don't need to waste time on.

Not excluding everything useless to the spiders has never inflicted any penalty in the SERPs for any site that I am aware of. Keep it simple.


All times are GMT. The time now is 12:59 PM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.00954 seconds
  • Memory Usage 1,725KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (2)bbcode_code_printable
  • (1)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (5)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 
Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 
Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete