vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vB4 General Discussions (https://vborg.vbsupport.ru/forumdisplay.php?f=251)
-   -   please verify robots.txt - much appreciated (https://vborg.vbsupport.ru/showthread.php?t=275709)

JIMRANGERS 12-20-2011 09:36 AM
please verify robots.txt - much appreciated
 
Quote:
User-agent: Baiduspider
Disallow: /

User-agent: BoardTracker
Disallow: /

User-agent: Gigabot
Disallow: /

User-agent: Twiceler
Disallow: /

User-agent: Slurp
Crawl-delay: 2

User-agent: msnbot
Crawl-delay: 2

User-agent: *
Disallow: *.js
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /images/
Disallow: /ajax.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /global.php
Disallow: /image.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /post.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php

Sitemap: http://yoursitessitemap.php

thanks in advance, was given this robot.txt from other vbulletin site but im not sure its correct as lots of URLs are being locked out according to google. Is this good or bad?

JIMRANGERS 12-21-2011 06:05 PM
sorry if its a bet technical, but cant find any answers on any forums

Macsee 12-21-2011 09:35 PM
Have you tried any of the robots.txt checking tools e.g. tool.motoricerca.info/robots-checker.phtml

You could also search in Google for robots.txt validator, there are several good ones.

On the face of it, yours looks OK. I usually also add

Disallow: /referrers.php

and, for SEO reasons

Disallow: /archive/

JIMRANGERS 12-25-2011 10:28 AM
Merry xmas all;

can someone explain why the /showpost.php should not be allowed, Im baffled on that one.

Boofo 12-25-2011 11:10 AM
Quote:
Originally Posted by Macsee (Post 2279854)
Have you tried any of the robots.txt checking tools e.g. tool.motoricerca.info/robots-checker.phtml

You could also search in Google for robots.txt validator, there are several good ones.

On the face of it, yours looks OK. I usually also add

Disallow: /referrers.php

and, for SEO reasons

Disallow: /archive/
There is no referrers.php file in 4.1.8.


All times are GMT. The time now is 01:22 PM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01052 seconds
  • Memory Usage 1,719KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (2)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (5)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 
Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 
Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete