OK, I fixed the xml file. I'm assuming that it won't update the previous crawls in the database?

nah it wont... thats why i been asking for a update button or something... if your good with sql then you can edit it yourself.. thats what i did... or you can clean them out from sql and let them crawl again

I'm just going to wait for one to be recognized so I know the xml format is working and then I'll empty the table.

Thanks.

no problem :)

That's coming... I haven't had time to get much of anything done lately, but it's on the list of things that will be in the next version.

yeah that would be nice... i cant wait to see the new version :)

Hmmm, they are still not being recognized by my site. This is what I have at the top of my file:

Does this look like yours?

Thanks.

here is mine... just upload and overwrite it or use it as a example... im going to update mine in a few days as i have to add some new bots in it...

That worked great! Thanks again.

no problem glad i could help...

Good hack, installed.

What does everyone recommend I do with this nasty beast:

Unknown Spider
Mozilla/5.0 (000000000; 0; 000 000 00 0 000000; 00000; 0000000000) 00000000000000 000000000000000

82.48.249.204
82.56.186.64

Is there an ID for it? If it won't identify itself should I just start blocking the IPs? Can I restrict them in my robots.txt?

Thanks!

personally im about to remove this hack... it has high queries and it sucks they dont update...

Its pretty stupid that it won't update old records in the DB, and since it uses so many queries it is obviously coded poorly.
But won't the high number of queries on affect the server when you load the spider page? I only allow admins to load it, so people shouldn't be hitting it all the time.

yeah they affect the server load... im removing it since i dont think it will be updated anytime soon... if it is then i may reinstall but as of right now it sucks editing from sql or having 500 bots unknown

But it only affects the server when you actually load the page right? Its not an ongoing thing with cron or anything is it?

It's not "stupid" that it doesn't update, the installation instructions say to update your spiders xml file BEFORE you install this. If you do that, then you won't have anything to update, as it'd be using the latest data.

The only time there should be a lot of queries is when it hasn't "rolled up" the data recently. After that first hit the number of queries should drop.

That may be the stupidiest thing I've ever heard.

Look, people, I wrote this thing for myself. I posted it here thinking that other people may benefit from it. If you want to use it, use it. If you don't, then don't. But don't go bashing it or my code. If you think you can do better, then write your own.

I will update it when I can, right now my top priority is finding a new job so I can support my family. Updating the the freebie hacks that I write when the mood strikes me aren't even in the top 10 of my list of priorities right now.

{edit}
One other thing... The first version that I put out deciphered the spider data when the user viewed the spiders page. If the spiders xml file had been updated after the data was collected it would be interpreted properly. But that version had way too many queries and everybody complained about it. Now the spider data is decoded when the spider visits and it's written to the database. The spiders page just collates all of that data and doesn't do any deciphering.

This hack is great bro.The only real issue i see with it is that sometimes the date and time doesnt get updated on a fresh spider visit to the forum.Sometimes the date reflects the true last visit and sometimes it doesnt.That is about the only thing i wish could be fixed up.Other than that,this hack is excellent and dont worry about harsh opinions of it,i cant wait to see the next update

Mike,

I appreciate this hack, it gives me good visibility to what the spiders are actually doing. I'm not a very good coder so I can't improve on your work.

My previous message was just stating that 58 queries is an extraordinarily large amount for such a simple page. It was my impression that the data would be regenerated when the spider page was loaded only, now I realize that its doing a lot of work each time a spider hits the site.

Thanks again.

great hack, but there lays a very crtitcal problem for me at least. When viewing the page.

Page generated in 0.59078 seconds with 50 queries [Server Loads: 0.41 0.22 : 0.10]

Thats quite alot of quires. The more spiders it fetches, the more queires this adds.

This hack killed my server today, was working fine today but it's messed my SQL and turned the server down.
I got like hundreds of e-mails like this one:

Database error in vBulletin 3.5.4:

Invalid SQL:
INSERT INTO spider_watcher (nice_bot, bot, ip_address, page, type, info, timestamp) VALUES ('Unknown Spider','Mozilla/4.0 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)','1198658994','index','','', NOW());

MySQL Error : Duplicate entry '431497' for key 1
Error Number : 1062
Date : Sunday, April 30th 2006 @ 08:17:10 PM

I don't think this error was caused by this hack... The key that it's complaining about is an auto_increment field, as far as I know there's no way to force it to try to insert a duplicate key like this is reporting. And the sql line shown above does not have anything about the key field that it's complaining about.

True, and I don't see any way to reduce those queries right now. The only thing I can think of to do is to run a job that will occasionaly "roll up" the spider data, and then when you view the spiders page it will only display data that's already been collated. But then the data displayed won't always be up to date. If anybody else has any ideas, I'm open to suggestions.

Well as soon as I removed the hack it started working fine...

I started to get the same e-mails again. Can you tell me exactly what this hack modifies on my board so I can look if anything is left after the uninstall?
Thanks

If you used the un-install function, it should have removed everything.

It creates a few plug-ins:
Spider Watcher
Spiders Location (Part 1)
Spiders Location (Part 2)
Spider Watcher Template Group

and it creates a few template in the "Spider Watcher Templates" group.

It's not a high traffic page. A lot of users jump onto the 'queries bad' bandwaggen a little to quickly. Queries make the vb world run.

Thanks mate, I stopped getting the e-mails and sever is running just fine.:cool:

nice hack, thanks.

*installed

With or without this hack installed?

i've been running this hack for a while, really nice work there mate...
and with no dramas ;)

Without.

Should it work instantly?

Really..Really strange...

its working SMOOTHLY in our board.. ..why u facing probs...

gez theres another thing..that s creating the pobs..
do u use "User access based on post count" hack?if yes..try w/o post count hack...

I uploaded the new spider list from the given link, I replaced it with the old one in my boards, but i still get unknown spider ALL the time.

http://www.total-clan.com/forums/spiders.php

hello anyone?

did u update ur spider list.xml properly ?????
get that from vbulletin.com

What did I say in my first post? lol:) I updated it:)

<a href="http://www.vbulletin.com/forum/showpost.php?p=565415&postcount=12" target="_blank">http://www.vbulletin.com/forum/showp...5&postcount=12</a>

Isnt this the list we are talking about? spiders.xml file?

It sounds like you are having the same problem that I had. The format for the spider file is not the same as what VB expects. Read back a few pages in this thread and there is another copy of the spider file you can use to get started.