vb.org Archive - perfect robots.txt for vb 3.7?

Page 1 of 3

Show 40 post(s) from this thread on one page

vb.org Archive (https://vborg.vbsupport.ru/index.php)

- vB3 General Discussions (https://vborg.vbsupport.ru/forumdisplay.php?f=111)

- - perfect robots.txt for vb 3.7? (https://vborg.vbsupport.ru/showthread.php?t=173206)

logicuk

03-16-2008 07:23 PM

perfect robots.txt for vb 3.7?

Does anyone care to share the perfect robots.txt for vb 3.7

here is what i have

Code:

User-agent: *

Disallow: /admincp/

Disallow: /archive/

Disallow: /backup/

Disallow: /announcement.php

Disallow: /calendar.php

Disallow: /cron.php

Disallow: /editpost.php

Disallow: /faq.php

Disallow: /joinrequests.php

Disallow: /login.php

Disallow: /member.php

Disallow: /misc.php

Disallow: /modcp/

Disallow: /moderator.php

Disallow: /newreply.php

Disallow: /newthread.php

Disallow: /online.php

Disallow: /printthread.php

Disallow: /private.php

Disallow: /profile.php

Disallow: /register.php

Disallow: /search.php

Disallow: /sendmessage.php

Disallow: /showgroups.php

Disallow: /showpost.php

Disallow: /subscription.php

Disallow: /subscriptions.php

Disallow: /threadrate.php

Disallow: /usercp.php

Boofo

03-16-2008 07:54 PM

You should have that in the domain root.

Bradley_Wint

03-17-2008 03:25 AM

don't mind me sounding like a newbie here but i added the content to a robots.txt file, upped it to my server html root folder like 8 hrs ago..now when i check my who's online list, i still see google bots murdering my calender.php, login.php...etc...

anything i did wrong?

blike2

03-17-2008 03:38 AM

Google only checks robots.txt every day or two, I think. Give it a little time.

Bradley_Wint

03-17-2008 03:39 AM

Quote:

Originally Posted by blike2 (Post 1466717)

Google only checks robots.txt every day or two, I think. Give it a little time.

thanks mate :)

SEOvB

03-17-2008 04:42 AM

once every 24hrs :)

Bradley_Wint

03-17-2008 04:45 AM

yes apparently it worked like 5 minutes after i posted... only the important files like forumdisplay, showpost, showthread etc we being scanned. thanks for this mate. cheers. :)

jambo_1969

03-17-2008 05:28 AM

Make sure you add in

User-agent: Fasterfox
Disallow: /

I managed to catch a thread on my board where people were discussing have Fasterfox set as turbo charged...my loads went through the roof !!

Boofo

03-17-2008 05:31 AM

Can we combine these 2?

Quote:

User-Agent: Googlebot-Image
Disallow: /

User-agent: Fasterfox
Disallow: /

Like this?

Quote:

User-Agent: Googlebot-Image
User-agent: Fasterfox
Disallow: /

jambo_1969

03-17-2008 05:36 AM

Quote:

Originally Posted by Boofo (Post 1466772)

Can we combine these 2?

Like this?

Not sure...

This is how mine looks (part of it anyway)

Code:

User-agent: Fasterfox

Disallow: /

User-agent: * 

Disallow: /.k.k.k./

All times are GMT. The time now is 11:38 AM.

Page 1 of 3

Show 40 post(s) from this thread on one page

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.01070 seconds Memory Usage 1,736KB Queries Executed 10 (?)
More Information
Template Usage: (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (2)bbcode_code_printable (4)bbcode_quote_printable (1)footer (1)gobutton (1)header (1)headinclude (6)option (1)pagenav (1)pagenav_curpage (2)pagenav_pagelink (1)post_thanks_navbar_search (1)printthread (10)printthreadbit (1)spacer_close (1)spacer_open Phrase Groups Available: global postbit showthread	Included Files: ./printthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/class_bbcode_alt.php ./includes/class_bbcode.php ./includes/functions_bigthree.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete printthread_start pagenav_page pagenav_complete bbcode_fetch_tags bbcode_create bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete printthread_post printthread_complete
Messages: