[DWard]

Can somebody make an updated version of the spiders list??

heres a good list:

Spider Identification Strings
===========================================
ABCdatos
Ahoy!
ia_archiver
AlkalineBOT
http://www.almaden.ibm.com/cs/crawler
EMC Spider
Anthill
Aport
Arachnophilia
Araneo
ArchitextSpider
arks/1.0
Ask Jeeves
ASpider
ATN_Worldwide
Atomz
AURESYS
BackRub
Baiduspider
bbot
Big Brother
BigmirSpider
Bjaaland
BlackWidow
Board Reader
BoardViewer
borg-bot
BSpider
CACTVS Chemistry Spider
Calif
Checkbot
ChristCrawler.com
www.cienciaficcion.net
CMC/0.01
combine
ComputingSite Robi/1.0
conceptbot
CoolBot
Cusco
CyberSpyder
DesertRealm.com
Deweb
Die Blinde Kuh
dienstspider
Digger/1.0 JDK/1.3.0
Digimarc WebReader
Digimarc CGIReader
DIIbot
grabber
DNAbot/1.0
DragonBot/1.0 libwww/5.0
DWCP/2.0
LWP::
EbiNess/0.01a
EIT-Link-Verifier-Robot/0.2
elfinbot
Emacs-w3/v[0-9\.]+
esther
Evliya Celebi
FAST Enterprise Crawler 6
FastCrawler
FelixIDE
ESIRover
fido
Fish-Search-Robot
Mozilla/4.0 (compatible: FDSE robot)
fouineur.9bit.qc.ca
Freecrawl
FunnelWeb
gammaSpider
gazz
gcreep
GetterroboPlus
GetURL.rexx
Gigabot
Girafabot
Golem
Googlebot
Mediapartners-Google
griffon
Gromit
http://grub.org
Gulper Web Bot
H?m?h?kki
havIndex
HeinrichderMiragoRobot
HenryTheMiragoRobot
HKU WWW Robot
Hometown
htdig
AITCSRobot
HTMLgobble
I Robot
iajaBot
IBM_Planetwide
IlTrovatore-Setaccio
image.kapsi.net
Mozilla 3.01 PBWF (Win95)
IncyWincy
Informant
InfoSeek Robot
Infoseek Sidewinder
InfoSpiders
INGRID
slurp@inktomi
inspectorwww
IAGENT
Internet Cruiser Robot
sharp-info-agent
InternetLinkAgent
Iron33
IsraeliSearch
JavaBee
JBot
JCrawler
JoBo
Jobot
JoeBot
jumpstation
Katipo
KDD-Explorer
KIT-Fireball
KO_Yappo_Robot
LabelGrab
larbin
legs
Linkidator
LinkScan Server
LinkWalker
Lockon
logo.gif crawler
Lycos
Magpie
marvin/infoseek
M/3.8
MediaFox
MerzScope
METASpider
MetaGer-LinkChecker
MindCrawler
UdmSearch
moget
MOMspider
Monster
Motor
MSNBOT
MuscatFerret
MwdSearch
NPBot
NEC-MeshExplorer
Nederland.zoek
NetCarta CyberPilot Pro
NetMechanic
NetScoop
newscan-online
NHSEWalker
Nomad
Gulliver
explorersearch
Occam
Openbot
Openfind
Orbsearch
PackRat
PageBoy
ParaSite
Patric
PEGASUS
PerlCrawler/1.0 Xavatoria/2.0
PGP-KA
Duppies
phpdig
PiltdownMan
Pimptrain's robot
Pioneer
PlumtreeWebAccessor
Poppi
gestaltIconoclast
PortalJuice.com
PortalBSpider
psbot
StackRambler
Raven
Resume Robot
Road Runner: ImageScape Robot
RHCS
Robbie
RoboCrawl
Robofox
Robot du CRIM 1.0a
Robozilla
Roverbot
RuLeS
SafetyNet Robot
Scooter
SearchNZ
searchprocess
Senrigan
SG-Scout
Shagseeker
Shai'Hulud
SimBot/1.0
ssearcher100
Site Valet
SiteTech-Rover
aWapClient
SLCrawler
Sleek Spider
ESISmartSpider
Snooper
Solbot
www.entireweb.com/speedy.html
SpiderBot
spiderline
SpiderMan
SpiderView
mouse.house
suke
suntek
Black Widow
Tarantula
tarspider
dlw3robot
TechBOT
Templeton
JubiiRobot
NorthStar
w3index
Peregrinator-Mathematics
TITAN
TitIn
TLSpider
turnitinbot
UCSD-Crawler
urlck
URL Spider Pro
Valkyrie
Verticrawl
Victoria
vision-search
Voyager
VWbot_K
W3M2
w3mir
w@pSpider
appie
CrawlPaper
root
WebMoose
WebBandit
WebCatcher
WebCopy
WebFetcher
weblayers
WebLinker
wlm
WebQuest
WebReaper
webs@recruit.co.jp
WOLP
webvac
webwalk
WebWalker
WebWatch
Wget
whatUseek_winona
Hazel's Ferret Web hopper
wired-digital-newsbot
zyborg
WWWC
WWWWanderer
TECOMAC-Crawler
XGET
cosmos
Yahoo! Slurp
Yahoo-VerticalCrawler
YahooFeedSeeker
Yandex



Spider Identification Description
===========================================
ABCdatos BotLink
Ahoy!
Alexa
Alkaline
Almaden Crawler
ananzi
Anthill
Aport
Arachnophilia
Araneo
ArchitextSpider
arks
AskJeeves
ASpider
ATN Worldwide
Atomz.com
AURESYS
BackRub
Baiduspider
BBot
Big Brother
Bigmir
Bjaaland
BlackWidow
BoardReader
BoardViewer
Borg-Bot
BSpider
CACTVS Chemistry
Calif
Checkbot
ChristCrawler.com
cIeNcIaFiCcIoN.nEt
CMC/0.01
Combine System
ComputingSite Robi/1.0
Conceptbot
CoolBot
Cusco
CyberSpyder
Desert Realm
DeWeb(c)
Die Blinde Kuh
DienstSpider
Digger
Digimarc MarcSpider
Digimarc Marcspider/CGI
Digital Integrity Robot
Direct Hit Grabber
DNAbot
DragonBot
DWCP (Dridus' Web Cataloging Project)
e-collector
EbiNess
EIT Link Verifier Robot
ELFINBOT
Emacs-w3 Search Engine
Esther
Evliya Celebi
FAST / AlltheWeb
FastCrawler
Felix IDE
FetchRover
fido
Fish search
Fluid Dynamics
Fouineur
Freecrawl
FunnelWeb
gammaSpider
gazz
GCreep
GetterroboPlus Puu
GetURL
Gigabot
Girafabot
Golem
Google
Google AdSense
Griffon
Gromit
Grub Client
Gulper Bot
H?m?h?kki
havIndex
HeinrichderMiragoRobot
HenryTheMiragoRobot
HKU WWW Octopus
Hometown
ht://Dig
HTML Index
HTMLgobble
I, Robot
iajaBot
IBM_Planetwide
IlTrovatore-Setaccio
image.kapsi.net
Imagelock
IncyWincy
Informant
InfoSeek Robot 1.0
Infoseek Sidewinder
InfoSpiders
Ingrid
Inktomi
Inspector Web
IntelliAgent
Internet Cruiser
Internet Shinchakubin
InternetLinkAgent
Iron33
Israeli-search
JavaBee
JBot
JCrawler
JoBo
Jobot
JoeBot
JumpStation
Katipo
KDD-Explorer
KIT-Fireball
KO_Yappo_Robot
LabelGrabber
larbin
legs
Link Validator
LinkScan
LinkWalker
Lockon
logo.gif
Lycos
Magpie
marvin/infoseek
Mattie
MediaFox
MerzScope
META
MetaGer
MindCrawler
mnoGoSearch
moget
MOMspider
Monster
Motor
MSNBot
Muscat Ferret
Mwd.Search
NameProtect
NEC-MeshExplorer
Nederland.zoek
NetCarta WebMap
NetMechanic
NetScoop
newscan-online
NHSE Web Forager
Nomad
Northern Light
nzexplorer
Occam
Openbot
Openfind data gatherer
Orb Search
Pack Rat
PageBoy
ParaSite
Patric
pegasus
PerlCrawler 1.0
PGP Key Agent
Phantom
PhpDig
PiltdownMan
Pimptrain.com's
Pioneer
PlumtreeWebAccessor
Poppi
Popular Iconoclast
Portal Juice
PortalB Spider
psbot
Rambler
Raven Search
Resume Robot
Road Runner: The ImageScape Robot
RoadHouse Crawling System
Robbie the Robot
RoboCrawl
RoboFox
Robot Francoroute
Robozilla
Roverbot
RuLeS
SafetyNet
Scooter
SearchNZ
SearchProcess
Senrigan
SG-Scout
ShagSeeker
Shai'Hulud
Simmany Robot Ver1.0
Site Searcher
Site Valet
SiteTech-Rover
Skymob.com
SLCrawler
Sleek
Smart Spider
Snooper
Solbot
Speedy Spider
SpiderBot
Spiderline Crawler
SpiderMan
SpiderView(tm)
spider_monkey
Suke
suntek search engine
TACH Black Widow
Tarantula
tarspider
Tcl W3 Robot
TechBOT
Templeton
The Jubii
The NorthStar Robot
The NWI Robot
The Peregrinator
TITAN
TitIn
TLSpider
Turnitin.com
UCSD Crawl
URL Check
URL Spider Pro
Valkyrie
Verticrawl
Victoria
vision-search
Voyager
VWbot
W3M2
w3mir
w@pSpider
Walhello appie
WallPaper
Web Core / Roots
Web Moose
WebBandit
WebCatcher
WebCopy
webfetcher
weblayers
WebLinker
Weblog Monitor
WebQuest
WebReaper
webs
WebStolperer
WebVac
webwalk
WebWalker
WebWatch
Wget
whatUseek Winona
Wild Ferret Web Hopper
Wired Digital
WiseNut
WWWC
WWWWanderer
X-Crawler
XGET
XYLEME Robot
Yahoo! Slurp
Yahoo-VerticalCrawler
YahooFeedSeeker
Yandex