vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vB3 General Discussions (https://vborg.vbsupport.ru/forumdisplay.php?f=111)
-   -   robots.txt help (https://vborg.vbsupport.ru/showthread.php?t=188216)

Chachacha 08-15-2008 01:12 PM
robots.txt help
 
Hello all. I have been doing a bit of research here on vb.org about robots.txt and have added it to my site... but its been 3 days and nothing has changed. My biggest problem is Yahoo! Slurp Spider. There are constantly multiple instances of it on my Who's Online page and they are constantly spidering my newthread.php and other pages that I don't want them wasting their time on. Google is a good boy, it only appears once at any given time in my WOL and it seems to crawl pages that I'd like it to crawl.

I've read that the robots.txt will fix this problem... but nothing has changed in 3 days. Isn't it supposed to take effect within 24 hrs?

I chmodded it to 755 is that correct? If not, what should I chmod it to?

Heres an exact copy of my robots.txt. Tell me if its correct...

Code:
User-agent: Slurp
Crawl-delay: 60
User-agent: Fasterfox
Disallow: /
User-agent: *
Disallow: /forums/admincp/
Disallow: /forums/clientscript/
Disallow: /forums/cpstyles/
Disallow: /forums/customavatars/
Disallow: /forums/customprofilepics/
Disallow: /forums/images/
Disallow: /forums/modcp/
Disallow: /forums/ajax.php
Disallow: /forums/attachment.php
Disallow: /forums/calendar.php
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/global.php
Disallow: /forums/image.php
Disallow: /forums/inlinemod.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newattachment.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/showpost.php
Disallow: /forums/subscription.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/usernote.php
Disallow: /forums/credits.php
Disallow: /forums/arcade.php
Disallow: /forums/vbimagehost.php

Lynne 08-15-2008 03:04 PM
The yahoo slurp! spider will take several days, if not a couple of weeks, until it starts obeying your robots.txt file. But it will finally do so, you just need to be patient.

Chachacha 08-15-2008 03:14 PM
Quote:
Originally Posted by Lynne (Post 1599302)
The yahoo slurp! spider will take several days, if not a couple of weeks, until it starts obeying your robots.txt file. But it will finally do so, you just need to be patient.
Thanks for your reply. One more question. In the part of the robots.txt file that says--->

User-agent: Slurp
Crawl-delay: 60

...Should I change "Slurp" to "Yahoo! Slurp Spider"?

Lynne 08-15-2008 03:52 PM
According to this link, you just use Slurp. http://help.yahoo.com/l/us/yahoo/sea.../slurp-03.html

Chachacha 08-15-2008 04:15 PM
Quote:
Originally Posted by Lynne (Post 1599355)
According to this link, you just use Slurp. http://help.yahoo.com/l/us/yahoo/sea.../slurp-03.html
You've been most helpful. Thank you very much.

Chachacha 08-29-2008 07:17 PM
Its been 2 weeks and Yahoo! Slurp Spider is still visiting pages that I've disallowed in my robots.txt.

SEOvB 08-29-2008 07:46 PM
Quote:
Originally Posted by Chachacha (Post 1610099)
Its been 2 weeks and Yahoo! Slurp Spider is still visiting pages that I've disallowed in my robots.txt.
make sure the robots.txt is in your web root and not the forum root.

Make sure the paths are correct, ex if your forums are in domain.com/forums/ folder then your posted robots.txt is correct. If your forums are not in the domain.com/forums/ folder, then your robots.txt is incorrect.

Chachacha 08-29-2008 07:51 PM
Quote:
Originally Posted by FRDS (Post 1610116)
make sure the robots.txt is in your web root and not the forum root.

Make sure the paths are correct, ex if your forums are in domain.com/forums/ folder then your posted robots.txt is correct. If your forums are not in the domain.com/forums/ folder, then your robots.txt is incorrect.
I have 3 copies of my robots.txt. One is in the MAIN folder when (if using c/panel) you go to "File Manager"... then I have one in the "Public_html" folder... and one in the "forums" folder. Yes, the path is correct.

SEOvB 08-29-2008 07:53 PM
you dont need any other than the public_html folder, they aren't hurting anything being in different locations to my knowledge, but they are just useless.

If the paths are correct and the yahoo spider is still going insane, try to run your robots.txt thru a robots.txt generator or checker (many available free online) to make sure there are no errors.

After that, i dont have a clue of why Yahoo isn't picking it up.

Chachacha 08-29-2008 08:16 PM
Quote:
Originally Posted by FRDS (Post 1610121)
you dont need any other than the public_html folder, they aren't hurting anything being in different locations to my knowledge, but they are just useless.

If the paths are correct and the yahoo spider is still going insane, try to run your robots.txt thru a robots.txt generator or checker (many available free online) to make sure there are no errors.

After that, i dont have a clue of why Yahoo isn't picking it up.
Thank you. I will try that.

EDIT: There were errors. I should have had an empty line between each block of code. I fixed it. Hope that was the problem.


All times are GMT. The time now is 03:18 PM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.02051 seconds
  • Memory Usage 1,738KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)bbcode_code_printable
  • (5)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (10)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 
Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 
Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete