vb.org Archive
Page 1 of 2 1 2
Show 40 post(s) from this thread on one page

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   Modification Requests/Questions (Unpaid) (https://vborg.vbsupport.ru/forumdisplay.php?f=112)
-   -   A robots.txt for vB3 RC2 please? (https://vborg.vbsupport.ru/showthread.php?t=60011)

Spencer F. 01-10-2004 02:50 PM
A robots.txt for vB3 RC2 please?
 
The title says it all. Thanks.

Boofo 01-10-2004 03:45 PM
Here's what I use.

Code:
User-Agent: Googlebot-Image
Disallow: /
 
User-agent: *
Disallow: /forums/attachment.php
Disallow: /forums/newattachment.php
Disallow: /forums/avatar.php
Disallow: /forums/editpost.php
Disallow: /forums/login.php
Disallow: /forums/member.php
Disallow: /forums/member2.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/private2.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/search.php
Disallow: /forums/sendtofriend.php
Disallow: /forums/showgroups.php
Disallow: /forums/showpost.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/usernote.php
Disallow: /forums/sendmessage.php
Disallow: /forums/register.php
Disallow: /forums/subscription.php
Disallow: /forums/admincp/
Disallow: /forums/archive/
Disallow: /forums/clientscript/
Disallow: /forums/cpstyles/
Disallow: /forums/customavatars/
Disallow: /forums/images/
Disallow: /forums/includes/
Disallow: /forums/install/
Disallow: /forums/modcp/
Disallow: /forums/subscriptions/

Zachery 01-10-2004 05:17 PM
you can also change User-Agent: Googlebot-Image
to User-Agent: *

to block most spiders :)

Boofo 01-10-2004 05:43 PM
This is mainly for Google grabbing images from your site tying up your bandwidth. The one below that catches the rest of them. ;)

You really don't want to block them all from everything or you won't get any listings in the search engines for your site.

Zachery 01-10-2004 06:48 PM
Quote:
Originally Posted by Boofo
This is mainly for Google grabbing images from your site tying up your bandwidth. The one below that catches the rest of them. ;)

You really don't want to block them all from everything or you won't get any listings in the search engines for your site.
ah i didnt relize you had the whole list there, i BELIVE scott released somthing like this from vbulletin.com to block spiders and there is an updated version

Boofo 01-10-2004 07:26 PM
This is the updated version from Scott's earlier release. ;)

Elmer 01-11-2004 03:42 AM
Quote:
Originally Posted by Boofo
This is the updated version from Scott's earlier release. ;)
What are robots? how can I use this:disappointed:

[high]* Elmer want to know a lil more[/high]

Zachery 01-11-2004 04:13 AM
Quote:
Originally Posted by Elmer
What are robots? how can I use this:disappointed:

[high]* Elmer want to know a lil more[/high]
robots are another name for spiders

which are what search engins such as google and yahoo use to index your site ;D

Dave-ahfb 01-11-2004 02:50 PM
Yahoo does not spider sites, unless you are speaking of Inktomi who Yahoo now owns.

Dave

Zachery 01-11-2004 02:53 PM
Quote:
Originally Posted by Dave-ahfb
Yahoo does not spider sites, unless you are speaking of Inktomi who Yahoo now owns.

Dave
yes they do


All times are GMT. The time now is 01:36 PM.
Page 1 of 2 1 2
Show 40 post(s) from this thread on one page

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01692 seconds
  • Memory Usage 1,736KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)bbcode_code_printable
  • (4)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (1)pagenav_pagelink
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (10)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 
Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 
Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • pagenav_page
  • pagenav_complete
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete