vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vB3 General Discussions (https://vborg.vbsupport.ru/forumdisplay.php?f=111)
-   -   perfect robots.txt for vb 3.7? (https://vborg.vbsupport.ru/showthread.php?t=173206)

logicuk 03-16-2008 07:23 PM
perfect robots.txt for vb 3.7?
 
Does anyone care to share the perfect robots.txt for vb 3.7

here is what i have

Code:
User-agent: *
Disallow: /admincp/
Disallow: /archive/
Disallow: /backup/
Disallow: /announcement.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /faq.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /misc.php
Disallow: /modcp/
Disallow: /moderator.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /subscription.php
Disallow: /subscriptions.php
Disallow: /threadrate.php
Disallow: /usercp.php

Boofo 03-16-2008 07:54 PM
You should have that in the domain root.

Bradley_Wint 03-17-2008 03:25 AM
don't mind me sounding like a newbie here but i added the content to a robots.txt file, upped it to my server html root folder like 8 hrs ago..now when i check my who's online list, i still see google bots murdering my calender.php, login.php...etc...

anything i did wrong?

blike2 03-17-2008 03:38 AM
Google only checks robots.txt every day or two, I think. Give it a little time.

Bradley_Wint 03-17-2008 03:39 AM
Quote:
Originally Posted by blike2 (Post 1466717)
Google only checks robots.txt every day or two, I think. Give it a little time.
thanks mate :)

SEOvB 03-17-2008 04:42 AM
once every 24hrs :)

Bradley_Wint 03-17-2008 04:45 AM
yes apparently it worked like 5 minutes after i posted... only the important files like forumdisplay, showpost, showthread etc we being scanned. thanks for this mate. cheers. :)

jambo_1969 03-17-2008 05:28 AM
Make sure you add in

User-agent: Fasterfox
Disallow: /

I managed to catch a thread on my board where people were discussing have Fasterfox set as turbo charged...my loads went through the roof !!

Boofo 03-17-2008 05:31 AM
Can we combine these 2?

Quote:
User-Agent: Googlebot-Image
Disallow: /

User-agent: Fasterfox
Disallow: /
Like this?

Quote:
User-Agent: Googlebot-Image
User-agent: Fasterfox
Disallow: /

jambo_1969 03-17-2008 05:36 AM
Quote:
Originally Posted by Boofo (Post 1466772)
Can we combine these 2?



Like this?
Not sure...

This is how mine looks (part of it anyway)
Code:
User-agent: Fasterfox
Disallow: /
User-agent: *
Disallow: /.k.k.k./
?

SEOvB 03-17-2008 05:51 AM
Quote:
Originally Posted by Boofo (Post 1466772)
Can we combine these 2?



Like this?

yes you can

Boofo 03-17-2008 06:07 AM
Thank you, sir. ;)

jambo_1969 03-17-2008 06:09 AM
Was there ever anything to stop Google Web Accelerator ?

logicuk 03-17-2008 10:34 AM
yeah i would like to know that also.

punchbowl 03-17-2008 11:05 AM
is it a bit of a security issue telling everyone where your admincp is?

Boofo 03-17-2008 11:07 AM
I don't see how as it is the default directory name for vb.

punchbowl 03-17-2008 11:13 AM
Quote:
Originally Posted by Boofo (Post 1466925)
I don't see how as it is the default directory name for vb.
It's been suggested you don't use the default directory though just to provide an extra layer of security. I have mine in a non-default location, htaccessed and with no links anywhere to it on my site. How would the googlebot ever find the location?

Considering all the brute force tools available to script kiddies I say out of sight, out of mind.

Boofo 03-17-2008 11:26 AM
All they would have to do was know any names of the default files in the admincp directory and they could find the directory that way. So even that isn't failsafe.

punchbowl 03-17-2008 11:30 AM
Quote:
Originally Posted by Boofo (Post 1466940)
All they would have to do was know any names of the default files in the admincp directory and they could find the directory that way. So even that isn't failsafe.

how?

http://www.domain.com/forum/yabbadabbado/index.php

Boofo 03-17-2008 11:54 AM
If they can get on the server, they can find the directory. There is no foolproof way to stop someone who wants to get to it bad enough. All you can do is slow them down. But good luck with your theory. ;)

punchbowl 03-17-2008 12:15 PM
Quote:
Originally Posted by Boofo (Post 1466957)
If they can get on the server, they can find the directory. There is no foolproof way to stop someone who wants to get to it bad enough. All you can do is slow them down. But good luck with your theory. ;)
It's not my theory. I think it may have been suggested on here. (Edit: Maybe not. Mentioned here http://www.vbulletin.com/forum/showthread.php?t=194701 also remember this?

Code:
//        ****** PATH TO ADMIN & MODERATOR CONTROL PANELS ******
        //        This setting allows you to change the name of the folders that the admin and
        //        moderator control panels reside in. You may wish to do this for security purposes.
        //        Please note that if you change the name of the directory here, you will still need
)

I only allow people http access to my server. Apart from guessing I'm a big fan of the flintstones it does provide extra hoops that any hacker will have to jump through. Not theoretical hoops either : They have to guess another defacto password. If someone has access to my server through ssh or whatever then it's already too late.

From reading on here 90% of board hacks are through a browser simply putting admincp after the board url and guessing. A large percentage of these 'hacks' are inside jobs by disgruntled mods who presumably know their way around a standard installation.

Why exclude something by name when googlebot only follows links and the admin link will never appear for guests? Are you putting your admin cp in a sitemap too?! (j/k re sitemap!)

Either way it's not a major issue but I just don't see the point in listing it.

Brandon Sheley 03-17-2008 12:44 PM
Quote:
Originally Posted by jambo_1969 (Post 1466769)
Make sure you add in

User-agent: Fasterfox
Disallow: /

I managed to catch a thread on my board where people were discussing have Fasterfox set as turbo charged...my loads went through the roof !!
Does anyone have any info on this? Does it really work? What's the point?

I've never heard of this before, I'll add it if it helps, but I really hate messing with my robots.txt file, I don't want to stop the bots from me site :D

thanks
-Brandon

punchbowl 03-17-2008 12:49 PM
Quote:
Originally Posted by Loco.M (Post 1466995)
Does anyone have any info on this? I've never heard of this before

thanks
-Brandon
Yeah I use it. Worth it.

There's a few other download accelerators too but some allow surfers spoof their useragent which is unfortunate.

Phooey 05-10-2008 12:41 PM
Does the fasterfox addon in your robots.txt completely block anyone using fasterfox or just blocks the functionality of the add-on on the site?

s3nt3nc3d 05-21-2008 07:57 PM
@Phooey
I'd assume it just blocks the functionality of the add-on

I run multiple websites on my server...only one is semi-large and having issues with spiders hogging all the resources (crashing my database multiple times a day due to having too many connections). Should I upload that file into the root folder of that one site or should I modify it and upload it into my server root folder where my xxxx.com, yyyyy.com, zzzzz.com, etc folders are located for my individual websites? I'm using 3.6.9 (soon to upgrade to 3.6.10--not quite ready for the jump to 3.7 but eventually will do so), so I'm gonna have to modify this a little to suit my needs prolly.

Jase2 05-21-2008 08:41 PM
Quote:
Originally Posted by Bradley_Wint (Post 1466710)
don't mind me sounding like a newbie here but i added the content to a robots.txt file, upped it to my server html root folder like 8 hrs ago..now when i check my who's online list, i still see google bots murdering my calender.php, login.php...etc...

anything i did wrong?
Give it a while. But are you sure they don't have the red error icon next to them?

Here's what I use :
Code:
User-agent: *
Disallow: /forum/admincp/
Disallow: /forum/album.php
Disallow: /forum/announcement.php
Disallow: /forum/calendar.php
Disallow: /forum/cron.php
Disallow: /forum/editpost.php
Disallow: /forum/faq.php
Disallow: /forum/showgroups.php
Disallow: /forum/images/
Disallow: /forum/joinrequests.php
Disallow: /forum/login.php
Disallow: /forum/memberslist.php
Disallow: /forum/misc.php
Disallow: /forum/modcp/
Disallow: /forum/moderator.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/poll.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/register.php
Disallow: /forum/report.php
Disallow: /forum/search.php
Disallow: /forum/sendmessage.php
Disallow: /forum/showpost.php
Disallow: /forum/subscription.php
Disallow: /forum/subscriptions.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
Disallow: /forum/usernote.php

SEOvB 05-21-2008 08:47 PM
Heres more information on Robots.txt for vBulletin including some other files and folders you should block out.

After you upload it, it could take up to 24hrs for spiders to find and start to follow the rules of robots.txt most only download it once a day.

Boofo 05-21-2008 09:41 PM
Too bad that link doesn't work.

Jase2 05-22-2008 01:26 PM
It does work....


All times are GMT. The time now is 04:57 PM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01181 seconds
  • Memory Usage 1,800KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (4)bbcode_code_printable
  • (11)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (29)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 
Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 
Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete