549 spiders on my site

[dtv100]

here a copy of my robot:

User-agent: *
User-agent: Slurp
User-agent: Mediapartners-Google*

Disallow: /forums/referrers.php
Disallow: /forums/arcade.php
Disallow: /forums/ajax_cron.php
Disallow: /forums/ajax.php
Disallow: /forums/attachment.php
Disallow: /forums/calendar.php
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/global.php
Disallow: /forums/image.php
Disallow: /forums/inlinemod.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/member.php
Disallow: /forums/memberlist.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newattachment.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/post_thanks.php
Disallow: /forums/payment_gateway.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/rules.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/showgroups.php
Disallow: /forums/subscription.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/admin/
Disallow: /forums/infernoshout.php
Disallow: /forums/redbar
Disallow: /forums/chat
Disallow: /forums/members/
Disallow: /forums/member-*

Sitemap:http://www.domain.com/forums/sitemap_index.xml.gz

[kNeeLy]

so..Is it better to disallow a few of your links?

[Brandon Sheley]

Originally Posted by kNeeLy

so..Is it better to disallow a few of your links?

You don't need the spiders checking out the pages or content that isn't directly related to your site

The robots.txt file posted above is good.. Everyone should have a copy of that in their root domain.. Yousite.com/robots.txt

Replace the domain info for your own with regards to the sitemap

here is the one in my root
http://www.vbulletinsetup.com/robots.txt
and since my forum is a subdomain, I have one for it as well
http://forum.vbulletinsetup.com/robots.txt

[ilrglen]

This may be a stupid quest but what is the gz for in these links to your sitemap?

# BEGIN XML-SITEMAP-PLUGIN
Sitemap: http://www.vbulletinsetup.com/sitemap.xml.gz
Sitemap: http://forum.vbulletinsetup.com/sitemap_index.xml.gz
# END XML-SITEMAP-PLUGIN

User-agent: Mediapartners-Google*
Disallow:
User-agent: *

Disallow: /pics
Disallow: /wp-admin
Disallow: /wp-login
Disallow: /wp-register
Disallow: /2006/
Disallow: /2007/
Disallow: /2008/