vb.org Archive
Page 3 of 3 12 3
Show 40 post(s) from this thread on one page

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vB3 General Discussions (https://vborg.vbsupport.ru/forumdisplay.php?f=111)
-   -   Yahoo! Slurp Spider, Sucking Us Dry (https://vborg.vbsupport.ru/showthread.php?t=102566)

chriscoleman 05-25-2007 12:48 PM
<div align="center">Another way of shutting out SLURP is by using the noindex meta-tag. Yahoo SLURP obeys this command in the document's head, and the code inserted in between the head tags of your document is

<META NAME=”robots” CONTENT=”noindex”>

This snippet will ensure that that Yahoo SLURP does not index the document in the search engine database. Another useful command is the nofollow meta-tag. The code inserted is

<META NAME=”robots” CONTENT=”nofollow”>

This snippet ensures that the links on the page are not followed.</div>
I found this on an SEO site.

Chachacha 08-10-2008 04:46 PM
Quote:
Originally Posted by Chris-777 (Post 1247788)
User-agent: Slurp
Crawl-delay: 60

Has solved all of my woes. :) I'm not sure what's been up in the last week, but with any other CD I've had hundreds of them online. I've no idea WHY that works, or if it will work for someone else, but it's working for me. (knock on wood)

Edit: It does still ignore my disallows though. :\

On the upside, I had 13,947 indexed pages in Yahoo two weeks ago, and currently I have 48,000, albeit after a LOT of SEO work, but it does seem like they're updating their content a bit more rigorously.
Where do I put that? In my htaccess?

Lynne 08-10-2008 04:51 PM
Quote:
Originally Posted by Chachacha (Post 1595592)
Where do I put that? In my htaccess?
No, you put it in your robots.txt file. You might want to search the site for more info on this.

Chachacha 08-10-2008 04:54 PM
Quote:
Originally Posted by Lynne (Post 1595596)
No, you put it in your robots.txt file. You might want to search the site for more info on this.
Ohh ok, thanks!

Alfa1 08-10-2008 08:17 PM
I have 2500 slurp spiders online at any given time. It got way more than it used to after I increased the crawl delay to 10. This may be coincidence.

Yahoo does not use more bandwidth than Google though. Yahoo just needs more spiders/IP's. Very annoying if half of your online users are bots.

Tharos 08-14-2008 03:16 AM
I think they are really good for our website's rank, I don't understand why you want to have less of them :s

TerryMason 08-19-2008 03:21 PM
From my forum:
Quote:
Currently Active Users: 1168 (3 members, 29 guests and 1136 spiders)

View Who's Online
Most users ever online was 2,369, 08-14-2008 at 10:55 PM.
I'd love to just have the text changed. It looks like I've just overinflated my stats.

Tharos 09-16-2008 09:16 PM
Could you tell us how did you get so many spiders in your web? :p


All times are GMT. The time now is 01:34 PM.
Page 3 of 3 12 3
Show 40 post(s) from this thread on one page

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01164 seconds
  • Memory Usage 1,728KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (4)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (2)pagenav_pagelink
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (8)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 
Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 
Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • pagenav_page
  • pagenav_complete
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete