A robots.txt for vB3 RC2 please?

[Spencer F.]

The title says it all. Thanks.

[Boofo]

Here's what I use.

User-Agent: Googlebot-Image
Disallow: /

User-agent: *
Disallow: /forums/attachment.php
Disallow: /forums/newattachment.php
Disallow: /forums/avatar.php
Disallow: /forums/editpost.php
Disallow: /forums/login.php
Disallow: /forums/member.php
Disallow: /forums/member2.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/private2.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/search.php
Disallow: /forums/sendtofriend.php
Disallow: /forums/showgroups.php
Disallow: /forums/showpost.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/usernote.php
Disallow: /forums/sendmessage.php
Disallow: /forums/register.php
Disallow: /forums/subscription.php
Disallow: /forums/admincp/
Disallow: /forums/archive/
Disallow: /forums/clientscript/
Disallow: /forums/cpstyles/
Disallow: /forums/customavatars/
Disallow: /forums/images/
Disallow: /forums/includes/
Disallow: /forums/install/
Disallow: /forums/modcp/
Disallow: /forums/subscriptions/

[Zachery]

you can also change User-Agent: Googlebot-Image
to User-Agent: *

to block most spiders :)

[Boofo]

This is mainly for Google grabbing images from your site tying up your bandwidth. The one below that catches the rest of them. ;)

You really don't want to block them all from everything or you won't get any listings in the search engines for your site.

[Zachery]

This is mainly for Google grabbing images from your site tying up your bandwidth. The one below that catches the rest of them. ;)

You really don't want to block them all from everything or you won't get any listings in the search engines for your site.
ah i didnt relize you had the whole list there, i BELIVE scott released somthing like this from vbulletin.com to block spiders and there is an updated version

[Boofo]

This is the updated version from Scott's earlier release. ;)

[Elmer]

This is the updated version from Scott's earlier release. ;)What are robots? how can I use this:disappointed:

* Elmer want to know a lil more

[Zachery]

What are robots? how can I use this:disappointed:

* Elmer want to know a lil more
robots are another name for spiders

which are what search engins such as google and yahoo use to index your site ;D

[Dave-ahfb]

Yahoo does not spider sites, unless you are speaking of Inktomi who Yahoo now owns.

Dave

[Zachery]

Yahoo does not spider sites, unless you are speaking of Inktomi who Yahoo now owns.

Dave
yes they do

[Dave-ahfb]

yes they do

And the spider is? Maybe you can share a little more info.

Dave

[Zachery]

yahoo uses Inktomi which is the yahoo spider...

[Dave-ahfb]

I know Ink spiders, but as of yet this is not used on the Yahoo site. Can you show me a SERP on Yahoo which is Inktomi and not Google?

In the next few months, what you say is true. Yahoo will not be renewing with Google and will replace it with what we know as Inktomi. Currently, as far as I know, it is all Google or Yahoo directory.

Dave

[Zach]

just fyi I am sure all know this, but just in case - yahoo uses their own now :) (only posted this just in case someobody read this and didnt realize it was old)

[Zach]

Disallow: /forums/archive/

?Why?

[Boofo]

Disallow: /forums/archive/


?Why?
That shouldn't be in there.