vb.org Archive - Robot Text File

vb.org Archive (https://vborg.vbsupport.ru/index.php)

- vB3 General Discussions (https://vborg.vbsupport.ru/forumdisplay.php?f=111)

- - Robot Text File (https://vborg.vbsupport.ru/showthread.php?t=163194)

Chance4Today

11-20-2007 02:01 PM

Robot Text File

Where do you locate one and is this essential to have for your website? When I first got started on this I had a tech support and they told me to have one like this.

User-Agent: *
Disallow:

Now I have gotten submitted my site to a few places and one of them says that this isn't good to have and is not readable. So I guess my question is where do I find one that I should be using for vbulletin and my sites. Very confusing I have done a search in all forums but only come up with hacks to use. Any suggestions on how to get the right robot.text ? Also have been looking should it be robot.text or robots.text? Is there a difference.

Lynne

11-20-2007 02:47 PM

You would put it in your root folder - so it would be here - /home/yoursitename/public_html/robots.txt Here's the top portion of mine:

Code:

User-agent: *

Crawl-delay: 40

Disallow: /forums/announcement.php

Disallow: /forums/attachment.php

Disallow: /forums/calendar.php

Disallow: /forums/editpost.php

Disallow: /forums/member.php

So, you need to put the names of the files you don't want the spiders to crawl after the disallow. The crawl-delay works to only allow so many spiders of a certain type (like those yahoo slurp spiders!) onto the site at a time.

Chance4Today

11-20-2007 05:19 PM

OK so is that all I need to put in mine? I guess I am really young at all of this and have been searching the internet oh I might remember where one was let me look LOL

Lynne

11-20-2007 10:32 PM

Quote:

Originally Posted by Chance4Today (Post 1386378)

OK so is that all I need to put in mine? I guess I am really young at all of this and have been searching the internet oh I might remember where one was let me look LOL

Well, you want to edit yours to list only the files you do not want the spiders to get into. I have several files I don't need them to index, and so I list them in the robots text. I also have problems with the yahoo slurp spiders which is why I have the crawl-delay. So, you just need to taylor yours to fit your needs.

If you google robots.txt, you will come up with several sites that explain all about way the file does and how to use it.

snakes1100

11-20-2007 11:26 PM

yahoo slurp bots dont listen anyways, only way to kill the majority of them is to ban all but one of the ips they come from, if i recall they spam from about 12 ips, but yahoo could of added more :D

Lynne

11-21-2007 02:33 AM

Quote:

Originally Posted by snakes1100 (Post 1386552)

yahoo slurp bots dont listen anyways, only way to kill the majority of them is to ban all but one of the ips they come from, if i recall they spam from about 12 ips, but yahoo could of added more :D

It works for me now. It took about three or four weeks, but the robots.txt finally worked.

All times are GMT. The time now is 01:27 AM.

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.01132 seconds Memory Usage 1,725KB Queries Executed 10 (?)
More Information
Template Usage: (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (1)bbcode_code_printable (2)bbcode_quote_printable (1)footer (1)gobutton (1)header (1)headinclude (6)option (1)post_thanks_navbar_search (1)printthread (6)printthreadbit (1)spacer_close (1)spacer_open Phrase Groups Available: global postbit showthread	Included Files: ./printthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/class_bbcode_alt.php ./includes/class_bbcode.php ./includes/functions_bigthree.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete printthread_start bbcode_fetch_tags bbcode_create bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete printthread_post printthread_complete
Messages: