vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   Modification Requests/Questions (Unpaid) (https://vborg.vbsupport.ru/forumdisplay.php?f=112)
-   -   A robots.txt for vB3 RC2 please? (https://vborg.vbsupport.ru/showthread.php?t=60011)

Spencer F. 01-10-2004 02:50 PM
A robots.txt for vB3 RC2 please?
 
The title says it all. Thanks.

Boofo 01-10-2004 03:45 PM
Here's what I use.

Code:
User-Agent: Googlebot-Image
Disallow: /
 
User-agent: *
Disallow: /forums/attachment.php
Disallow: /forums/newattachment.php
Disallow: /forums/avatar.php
Disallow: /forums/editpost.php
Disallow: /forums/login.php
Disallow: /forums/member.php
Disallow: /forums/member2.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/private2.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/search.php
Disallow: /forums/sendtofriend.php
Disallow: /forums/showgroups.php
Disallow: /forums/showpost.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/usernote.php
Disallow: /forums/sendmessage.php
Disallow: /forums/register.php
Disallow: /forums/subscription.php
Disallow: /forums/admincp/
Disallow: /forums/archive/
Disallow: /forums/clientscript/
Disallow: /forums/cpstyles/
Disallow: /forums/customavatars/
Disallow: /forums/images/
Disallow: /forums/includes/
Disallow: /forums/install/
Disallow: /forums/modcp/
Disallow: /forums/subscriptions/

Zachery 01-10-2004 05:17 PM
you can also change User-Agent: Googlebot-Image
to User-Agent: *

to block most spiders :)

Boofo 01-10-2004 05:43 PM
This is mainly for Google grabbing images from your site tying up your bandwidth. The one below that catches the rest of them. ;)

You really don't want to block them all from everything or you won't get any listings in the search engines for your site.

Zachery 01-10-2004 06:48 PM
Quote:
Originally Posted by Boofo
This is mainly for Google grabbing images from your site tying up your bandwidth. The one below that catches the rest of them. ;)

You really don't want to block them all from everything or you won't get any listings in the search engines for your site.
ah i didnt relize you had the whole list there, i BELIVE scott released somthing like this from vbulletin.com to block spiders and there is an updated version

Boofo 01-10-2004 07:26 PM
This is the updated version from Scott's earlier release. ;)

Elmer 01-11-2004 03:42 AM
Quote:
Originally Posted by Boofo
This is the updated version from Scott's earlier release. ;)
What are robots? how can I use this:disappointed:

[high]* Elmer want to know a lil more[/high]

Zachery 01-11-2004 04:13 AM
Quote:
Originally Posted by Elmer
What are robots? how can I use this:disappointed:

[high]* Elmer want to know a lil more[/high]
robots are another name for spiders

which are what search engins such as google and yahoo use to index your site ;D

Dave-ahfb 01-11-2004 02:50 PM
Yahoo does not spider sites, unless you are speaking of Inktomi who Yahoo now owns.

Dave

Zachery 01-11-2004 02:53 PM
Quote:
Originally Posted by Dave-ahfb
Yahoo does not spider sites, unless you are speaking of Inktomi who Yahoo now owns.

Dave
yes they do

Dave-ahfb 01-11-2004 06:35 PM
Quote:
Originally Posted by Faranth
yes they do
And the spider is? Maybe you can share a little more info.

Dave

Zachery 01-11-2004 06:43 PM
yahoo uses Inktomi which is the yahoo spider...

Dave-ahfb 01-11-2004 07:09 PM
I know Ink spiders, but as of yet this is not used on the Yahoo site. Can you show me a SERP on Yahoo which is Inktomi and not Google?

In the next few months, what you say is true. Yahoo will not be renewing with Google and will replace it with what we know as Inktomi. Currently, as far as I know, it is all Google or Yahoo directory.

Dave

Zach 05-13-2004 04:13 PM
just fyi I am sure all know this, but just in case - yahoo uses their own now :) (only posted this just in case someobody read this and didnt realize it was old)

Zach 05-13-2004 04:16 PM
Disallow: /forums/archive/

?Why?

Boofo 05-13-2004 04:23 PM
Quote:
Originally Posted by Zach
Disallow: /forums/archive/


?Why?
That shouldn't be in there.


All times are GMT. The time now is 06:52 AM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.02103 seconds
  • Memory Usage 1,745KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)bbcode_code_printable
  • (6)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (16)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 
Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 
Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete