Yahoo! Slurp Spider, Sucking Us Dry

[chriscoleman]

<div align="center">Another way of shutting out SLURP is by using the noindex meta-tag. Yahoo SLURP obeys this command in the document's head, and the code inserted in between the head tags of your document is

<META NAME=”robots” CONTENT=”noindex”>

This snippet will ensure that that Yahoo SLURP does not index the document in the search engine database. Another useful command is the nofollow meta-tag. The code inserted is

<META NAME=”robots” CONTENT=”nofollow”>

This snippet ensures that the links on the page are not followed.</div>
I found this on an SEO site.

[Chachacha]

Originally Posted by Chris-777

User-agent: Slurp
Crawl-delay: 60

Has solved all of my woes. I'm not sure what's been up in the last week, but with any other CD I've had hundreds of them online. I've no idea WHY that works, or if it will work for someone else, but it's working for me. (knock on wood)

Edit: It does still ignore my disallows though. :\

On the upside, I had 13,947 indexed pages in Yahoo two weeks ago, and currently I have 48,000, albeit after a LOT of SEO work, but it does seem like they're updating their content a bit more rigorously.

Where do I put that? In my htaccess?

[Lynne]

Originally Posted by Chachacha

Where do I put that? In my htaccess?

No, you put it in your robots.txt file. You might want to search the site for more info on this.

[Chachacha]

Originally Posted by Lynne

No, you put it in your robots.txt file. You might want to search the site for more info on this.

Ohh ok, thanks!

[Alfa1]

I have 2500 slurp spiders online at any given time. It got way more than it used to after I increased the crawl delay to 10. This may be coincidence.

Yahoo does not use more bandwidth than Google though. Yahoo just needs more spiders/IP's. Very annoying if half of your online users are bots.

[Tharos]

I think they are really good for our website's rank, I don't understand why you want to have less of them :s

[TerryMason]

From my forum:

Currently Active Users: 1168 (3 members, 29 guests and 1136 spiders)

View Who's Online
Most users ever online was 2,369, 08-14-2008 at 10:55 PM.

I'd love to just have the text changed. It looks like I've just overinflated my stats.

[Tharos]

Could you tell us how did you get so many spiders in your web?