Spider Watcher

[mikelbeck]

thanx fot the update, argh all the activity is gone now. but thanks again

What version did you update from? There wasn't anything in this version to delete your old data.

Updated seems ok :)

Think I would prefer a popup window that can be used like being able to copy the IP this would make it easier to add the bots ip to a banned IP list if you didnt want it on your site and it was ignoring robot.txt.

I'll see what I can do with that.


Another little suggestion the Bot name could be a link to a search engine so you can quickly access links about a bot and its origins and behaviour.

If there's data in the spiders file about the URL, the spider name is linked to that URL.

[Zia]

Updated :).. !seen Zia :P.. Newayz thnx Mikelbeck..

lolz...u should have wait for me...


Grate old data didnt lost....Thnx MikelBech.


Micel plz wait for next release..i collect all the data of unknown spider from my site & from happyhourpub too. n i submiited this in vb.com n some other place to identify them properly n put them in the list.xml . After getting the update list , the next version will be more perfect.


Thanks

[2ndCz]

Thanks for the update. Love this hack!

Just wondering, the last activity date/time on my spider info seems to be off by like a day. The spider hits go up fine, but the date seems off to me.

My server time is +2 from me. So if that was the problem I think the time would be 2 hours ahead (not a day behind), wouldn't it?

Any ideas? Thanks.

[mikelbeck]

Micel plz wait for next release..i collect all the data of unknown spider from my site & from happyhourpub too. n i submiited this in vb.com n some other place to identify them properly n put them in the list.xml . After getting the update list , the next version will be more perfect.


Thanks

No update would be required to this hack if/when th vbulletin_spiders.xml file is updated. Just replace that file on your server, this hack will pick up the changes and will start reporting the updated spiders.

[mikelbeck]

Thanks for the update. Love this hack!

Just wondering, the last activity date/time on my spider info seems to be off by like a day. The spider hits go up fine, but the date seems off to me.

My server time is +2 from me. So if that was the problem I think the time would be 2 hours ahead (not a day behind), wouldn't it?

Any ideas? Thanks.
Are you using the latest version? Do you have a URL?

[Traxdata]

Cool idea, but there must be a bug or something.
After install I couldn't use "New Posts" function (search.php?do=getnew) - Site not found.
Note: "Search" function (search.php) worked fine.
I have disabled this hack and everything is fine now.
Tested on vB 3.5.2.

[2ndCz]

Are you using the latest version? Do you have a URL?

Yep. Downloaded the latest version today.

URL: http://forums.secondcitizen.com/spiders.php

[mikelbeck]

Cool idea, but there must be a bug or something.
After install I couldn't use "New Posts" function (search.php?do=getnew) - Site not found.
Note: "Search" function (search.php) worked fine.
I have disabled this hack and everything is fine now.
Tested on vB 3.5.2.
I can't imagine why that would happen... This hack has nothing to do with the search functions... Did it only happen when you did a "new posts" search or on any other pages?

[Robbed]

Afer uprading all the other spiders look like this

Unknown Spider
Gigabot/2.0/gigablast.com/spider.html

They all say unknown? How would I clear the data to start fresh?

[Traxdata]

I can't imagine why that would happen... This hack has nothing to do with the search functions... Did it only happen when you did a "new posts" search or on any other pages?

Yes, just with "New Posts" functions but now I tested this hack on vB 3.5.4 - works like charm!
Thank you very much !

[mikelbeck]

Yes, just with "New Posts" functions but now I tested this hack on vB 3.5.4 - works like charm!
Thank you very much !
Well, good to hear it's working for you!

[Robbed]

found it n/m

[MoveOver.cc]

I am getting the following error -

You have an error in your SQL syntax near 'START TRANSACTION' at line 1
Error Number : 1064

This happens when I type in the link /spiders.php.

Can anyone advise.

thanks

[DementedMindz]

is there a way to make it update the spider.php page? i added new spiders in the list but it would be nice if they werent on bottom no more and would refresh and go to the top... or do i have to clean the sql out???? maybe you can add a feature this way we can clean the list easier??? or update the list ....

[Brandon Sheley]

a option to clear list would be great :)

I like the new grouping

[DementedMindz]

any word??

[mikelbeck]

Sorry, I've been tied up lately and haven't had a change to reply.

MoveOver.cc: Open up spiders.php, search for this line:
$cleanup_data = $db->query_first("START TRANSACTION");

and change it to:
$cleanup_data = $db->query_first("BEGIN");

DementedMindz: Not at this time. Perhaps I'll add that to the next version.

Loco.M: I'll add something to the next version to allow you to empty your data.

[cudaxtreme]

a option to clear list would be great :)

I like the new grouping

I uninstall and reinstall the plugin to clear the list.

[nicedreams]

I got this error overnight:

Database error in vBulletin 3.5.4:

Invalid SQL:
INSERT INTO spider_watcher (nice_bot, bot, ip_address, page, type, info, timestamp) VALUES ('Unknown Spider','<script>window.open('http://www.medchecker.com/side.htm','_search')</script> (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)','1119861065','online','','', NOW());

MySQL Error : You have an error in your SQL syntax. Check the manual that corresponds to your MySQL server version for the right syntax to use near 'http://www.medchecker.com/side.htm','_search')</script> ( Error Number : 1064

Jim

[mcncyo]

I got this error overnight:

Database error in vBulletin 3.5.4:

Invalid SQL:
INSERT INTO spider_watcher (nice_bot, bot, ip_address, page, type, info, timestamp) VALUES ('Unknown Spider','<script>window.open('http://www.medchecker.com/side.htm','_search')</script> (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)','1119861065','online','','', NOW());

MySQL Error : You have an error in your SQL syntax. Check the manual that corresponds to your MySQL server version for the right syntax to use near 'http://www.medchecker.com/side.htm','_search')</script> ( Error Number : 1064

Jim

I also got this error a couple of days ago.

[tribedude]

I uploaded the newest spiderlist from vbulletin, then installed this product and it seems to be working however I am getting tons of "Unknown Spider' in bold but with an URL below that identifies what the spider is... Is it normal for it to say Unknow spider like that? How do i get it to dsiplay what spider it actually is instead of the URL?

[DementedMindz]

yeah i get alot like that too... i edit the list manually when i get that and then edit it in sql.... also was wondering do you think mikelbeck if you removed the ip lookup it would cut back on the sql queries? cause it does go high when you get like 50 and above spiders plus yahoo on there i have like 30 different ips alone from yahoo

[mikelbeck]

tribedude - if the spider's data isn't in the spiderlist, it's going to show up as "unknown".

DementedMindz - I don't think that the IP lookup has much to do with sql queries, the IP data is stored with the rest of the spider data, and it's "rolled up" along with everything else.

[DementedMindz]

so is there anyway to cut back on the sql queries? and also maybe make it where only certain usergroups can see it....

[Gottcha]

Warning: array_multisort(): Argument #1 is expected to be an array or a sort flag in /spiders.php on line 223

any help is appreciated ... thanks

[Trana]

I uploaded the newest spiderlist from vbulletin, then installed this product and it seems to be working however I am getting tons of "Unknown Spider' in bold but with an URL below that identifies what the spider is... Is it normal for it to say Unknow spider like that? How do i get it to dsiplay what spider it actually is instead of the URL?

Strange, I don't see it picking up ANY of the spiders. I have everything properly defined in my XML file (using the latest definitions), and it still sees Yahoo Slurp and Googlebot and "Unknown Spider".

[DementedMindz]

working great on my site... just needs to lower the queries a bit there still way high

[Trana]

working great on my site... just needs to lower the queries a bit there still way high

Would you mind posting the first few lines of your spider xml file? I got mine off the spider site but I don't think it is formatted correctly.

Thanks.

[DementedMindz]

yeah i changed my format around as it didnt seem like theres was like vbulletins...


this is the top..

<?xml version="1.0" encoding="ISO-8859-1"?>

<searchspiders>

and at the very bottom its closed like this:

</searchspiders>

<!-- CVS: $RCSfile: spiders_vbulletin.xml,v $ - $Revision: 1.2 $ -->

[DementedMindz]

im still adding bots to mine as i see them crawl... cause there is alot of bots they dont even have listed too

[Trana]

OK, I fixed the xml file. I'm assuming that it won't update the previous crawls in the database?

[DementedMindz]

nah it wont... thats why i been asking for a update button or something... if your good with sql then you can edit it yourself.. thats what i did... or you can clean them out from sql and let them crawl again

[Trana]

I'm just going to wait for one to be recognized so I know the xml format is working and then I'll empty the table.

Thanks.

[DementedMindz]

no problem :)

[mikelbeck]

nah it wont... thats why i been asking for a update button or something... if your good with sql then you can edit it yourself.. thats what i did... or you can clean them out from sql and let them crawl again
That's coming... I haven't had time to get much of anything done lately, but it's on the list of things that will be in the next version.

[DementedMindz]

yeah that would be nice... i cant wait to see the new version :)

[Trana]

Hmmm, they are still not being recognized by my site. This is what I have at the top of my file:

<?xml version="1.0" encoding="ISO-8859-1" ?>
<searchspiders>
<!DOCTYPE spiderlist (View Source for full doctype...)>
- <!--
$Id$
-->
- <spiderlist version="1.0">
- <spider ident="ABCdatos">
<name>ABCdatos BotLink</name>
<type>searchspider</type>
<info>http://www.robotstxt.org/wc/active/html/abcdatos.html</info>
<email>botlink+AEA-abcdatos.com</email>
- <addresses>
<address type="CIDR">217.126.0.0/18</address>
</addresses>
</spider>
- <spider ident="abot/">

Does this look like yours?

Thanks.

[DementedMindz]

here is mine... just upload and overwrite it or use it as a example... im going to update mine in a few days as i have to add some new bots in it...

[Trana]

That worked great! Thanks again.

[DementedMindz]

no problem glad i could help...

[The Notorious]

Good hack, installed.

[Trana]

What does everyone recommend I do with this nasty beast:

Unknown Spider
Mozilla/5.0 (000000000; 0; 000 000 00 0 000000; 00000; 0000000000) 00000000000000 000000000000000

82.48.249.204
82.56.186.64

Is there an ID for it? If it won't identify itself should I just start blocking the IPs? Can I restrict them in my robots.txt?

Thanks!

[DementedMindz]

personally im about to remove this hack... it has high queries and it sucks they dont update...

[Trana]

personally im about to remove this hack... it has high queries and it sucks they dont update...

Its pretty stupid that it won't update old records in the DB, and since it uses so many queries it is obviously coded poorly.
But won't the high number of queries on affect the server when you load the spider page? I only allow admins to load it, so people shouldn't be hitting it all the time.

[DementedMindz]

yeah they affect the server load... im removing it since i dont think it will be updated anytime soon... if it is then i may reinstall but as of right now it sucks editing from sql or having 500 bots unknown

[Trana]

But it only affects the server when you actually load the page right? Its not an ongoing thing with cron or anything is it?

[mikelbeck]

It's not "stupid" that it doesn't update, the installation instructions say to update your spiders xml file BEFORE you install this. If you do that, then you won't have anything to update, as it'd be using the latest data.

The only time there should be a lot of queries is when it hasn't "rolled up" the data recently. After that first hit the number of queries should drop.

and since it uses so many queries it is obviously coded poorly

That may be the stupidiest thing I've ever heard.

Look, people, I wrote this thing for myself. I posted it here thinking that other people may benefit from it. If you want to use it, use it. If you don't, then don't. But don't go bashing it or my code. If you think you can do better, then write your own.

I will update it when I can, right now my top priority is finding a new job so I can support my family. Updating the the freebie hacks that I write when the mood strikes me aren't even in the top 10 of my list of priorities right now.

{edit}
One other thing... The first version that I put out deciphered the spider data when the user viewed the spiders page. If the spiders xml file had been updated after the data was collected it would be interpreted properly. But that version had way too many queries and everybody complained about it. Now the spider data is decoded when the spider visits and it's written to the database. The spiders page just collates all of that data and doesn't do any deciphering.

[xStylezx]

This hack is great bro.The only real issue i see with it is that sometimes the date and time doesnt get updated on a fresh spider visit to the forum.Sometimes the date reflects the true last visit and sometimes it doesnt.That is about the only thing i wish could be fixed up.Other than that,this hack is excellent and dont worry about harsh opinions of it,i cant wait to see the next update

[Trana]

Mike,

I appreciate this hack, it gives me good visibility to what the spiders are actually doing. I'm not a very good coder so I can't improve on your work.

My previous message was just stating that 58 queries is an extraordinarily large amount for such a simple page. It was my impression that the data would be regenerated when the spider page was loaded only, now I realize that its doing a lot of work each time a spider hits the site.

Thanks again.

[Logikos]

great hack, but there lays a very crtitcal problem for me at least. When viewing the page.

Page generated in 0.59078 seconds with 50 queries [Server Loads: 0.41 0.22 : 0.10]

Thats quite alot of quires. The more spiders it fetches, the more queires this adds.

[The Notorious]

This hack killed my server today, was working fine today but it's messed my SQL and turned the server down.
I got like hundreds of e-mails like this one:

Database error in vBulletin 3.5.4:

Invalid SQL:
INSERT INTO spider_watcher (nice_bot, bot, ip_address, page, type, info, timestamp) VALUES ('Unknown Spider','Mozilla/4.0 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)','1198658994','index','','', NOW());

MySQL Error : Duplicate entry '431497' for key 1
Error Number : 1062
Date : Sunday, April 30th 2006 @ 08:17:10 PM

[mikelbeck]

This hack killed my server today, was working fine today but it's messed my SQL and turned the server down.
I got like hundreds of e-mails like this one:

Database error in vBulletin 3.5.4:

Invalid SQL:
INSERT INTO spider_watcher (nice_bot, bot, ip_address, page, type, info, timestamp) VALUES ('Unknown Spider','Mozilla/4.0 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)','1198658994','index','','', NOW());

MySQL Error : Duplicate entry '431497' for key 1
Error Number : 1062
Date : Sunday, April 30th 2006 @ 08:17:10 PM
I don't think this error was caused by this hack... The key that it's complaining about is an auto_increment field, as far as I know there's no way to force it to try to insert a duplicate key like this is reporting. And the sql line shown above does not have anything about the key field that it's complaining about.

[mikelbeck]

great hack, but there lays a very crtitcal problem for me at least. When viewing the page.

Page generated in 0.59078 seconds with 50 queries [Server Loads: 0.41 0.22 : 0.10]

Thats quite alot of quires. The more spiders it fetches, the more queires this adds.

True, and I don't see any way to reduce those queries right now. The only thing I can think of to do is to run a job that will occasionaly "roll up" the spider data, and then when you view the spiders page it will only display data that's already been collated. But then the data displayed won't always be up to date. If anybody else has any ideas, I'm open to suggestions.

[The Notorious]

I don't think this error was caused by this hack... The key that it's complaining about is an auto_increment field, as far as I know there's no way to force it to try to insert a duplicate key like this is reporting. And the sql line shown above does not have anything about the key field that it's complaining about.
Well as soon as I removed the hack it started working fine...

[The Notorious]

I started to get the same e-mails again. Can you tell me exactly what this hack modifies on my board so I can look if anything is left after the uninstall?
Thanks

[mikelbeck]

I started to get the same e-mails again. Can you tell me exactly what this hack modifies on my board so I can look if anything is left after the uninstall?
Thanks
If you used the un-install function, it should have removed everything.

It creates a few plug-ins:
Spider Watcher
Spiders Location (Part 1)
Spiders Location (Part 2)
Spider Watcher Template Group

and it creates a few template in the "Spider Watcher Templates" group.

[hambil]

True, and I don't see any way to reduce those queries right now. The only thing I can think of to do is to run a job that will occasionaly "roll up" the spider data, and then when you view the spiders page it will only display data that's already been collated. But then the data displayed won't always be up to date. If anybody else has any ideas, I'm open to suggestions.
It's not a high traffic page. A lot of users jump onto the 'queries bad' bandwaggen a little to quickly. Queries make the vb world run.

[The Notorious]

If you used the un-install function, it should have removed everything.

It creates a few plug-ins:
Spider Watcher
Spiders Location (Part 1)
Spiders Location (Part 2)
Spider Watcher Template Group

and it creates a few template in the "Spider Watcher Templates" group.
Thanks mate, I stopped getting the e-mails and sever is running just fine.:cool:

[Dr.Viggy]

nice hack, thanks.

*installed

[mikelbeck]

Thanks mate, I stopped getting the e-mails and sever is running just fine.:cool:
With or without this hack installed?

[kurtbarker]

i've been running this hack for a while, really nice work there mate...
and with no dramas ;)

[The Notorious]

With or without this hack installed?
Without.

[kofoid]

Should it work instantly?

[Zia]

Without.


Really..Really strange...

its working SMOOTHLY in our board. (http://www.golpo.net/forum/spiders.php). ..why u facing probs...

gez theres another thing..that s creating the pobs..
do u use "User access based on post count" hack?if yes..try w/o post count hack...

[maranello]

I uploaded the new spider list from the given link, I replaced it with the old one in my boards, but i still get unknown spider ALL the time.

http://www.total-clan.com/forums/spiders.php

[maranello]

hello anyone?

[Zia]

I uploaded the new spider list from the given link, I replaced it with the old one in my boards, but i still get unknown spider ALL the time.

http://www.total-clan.com/forums/spiders.php


did u update ur spider list.xml properly ?????
get that from vbulletin.com

[maranello]

What did I say in my first post? lol:) I updated it:)

[maranello]

<a href="http://www.vbulletin.com/forum/showpost.php?p=565415&postcount=12" target="_blank">http://www.vbulletin.com/forum/showp...5&postcount=12</a>

Isnt this the list we are talking about? spiders.xml file?

[Trana]

It sounds like you are having the same problem that I had. The format for the spider file is not the same as what VB expects. Read back a few pages in this thread and there is another copy of the spider file you can use to get started.

[maranello]

Alright, I did what you said, still no change.

[maranello]

Also, is there an easy way to test this to see it works? It sometimes takes days for new stuff to appear in that page.

[mikelbeck]

Also, is there an easy way to test this to see it works? It sometimes takes days for new stuff to appear in that page.

No, not really. Just gets lots of spiders to your pages. ;-)

As for the problems you're having, the only thing I can think of is that something is up with your spiders_vbulletin.xml file. You say you pulled down that latest file, but did you rename it to spiders_vbulletin.xml and place it in your includes/xml directory? Also, check the permissions on the file, it should be 644. If it's not chmod it to that.

[Snake]

Actually mikel, I am also having the exact, same problem. I've got many unknown spiders on my forums. Here have a look...

http://www.bonethugsforums.com/spiders.html

Do you have any idea what's wrong?

[maranello]

mike, i did what you said, still nothing, chmod is 644, and i renamed the file and its in correct location.

[mikelbeck]

Actually mikel, I am also having the exact, same problem. I've got many unknown spiders on my forums. Here have a look...

http://www.bonethugsforums.com/spiders.html

Do you have any idea what's wrong?
Yours is normal, the "unknown spiders" are ones that are identified as spiders but are not listed in the spiders_vbulletin.xml file.

[Zia]

we need a massive update of spiderlist.xml..

mikelbeck..the spiders.php page execute a HUGE SQL query..in our page its 416 query..:S ..do u have any idea to reduce sql query in next release???

[maranello]

someone needs to update the spiders.xml file.

[mikelbeck]

we need a massive update of spiderlist.xml..

mikelbeck..the spiders.php page execute a HUGE SQL query..in our page its 416 query..:S ..do u have any idea to reduce sql query in next release???
Yes, I have a couple of ideas. But I'm not going to be able to get to them for a little while...

[Zia]

Yes, I have a couple of ideas. But I'm not going to be able to get to them for a little while...

thnx man...hope u will impliment those idea to reduced sql query as earlier as possible..:)


TIA.

[maranello]

am I the only one having this unknown spider bot problem?

I dont have a SINGLE known spider in the list:

http://www.total-clan.com/forums/spiders.php

I have the LATEST xml file, I formatted it right, chmod is right, everything is right. What's wrong?

[MThornback]

99% of the time mine are unknown too...I used the spiders from hambil's BOL hack, and it gives me a known one every once in a blue moon...but the googlebot, MSNbot both show up...so it might be a step in the direction your after.

[blogtorank]

***UPGRADE NOTE***
When you upgrade from version 1.0.0 Beta 7 to 1.0.0 Beta 8 your existing spider data will be lost!


I almost posted where did they go however glad this is there!

[dwbro1]

Why is adv_index showing up in the list I thought this was replaced when I renamed it to index.php, or am I just loosing my mind.

[mikelbeck]

Why is adv_index showing up in the list I thought this was replaced when I renamed it to index.php, or am I just loosing my mind.

You could be losing your mind, but that's something you'll have to deal with on your own. ;-)

That page probably calls itself adv_index (in the code of the page), so that's what the spider watcher sees when the page is loaded. You can rename the file to whatever you want, but if you don't change the code it'll still have it's original name.

BTW - don't go changing the page's name if you don't know what you're doing... It's used in other places than just the file itself, if you just change it there you can screw other stuff up.

[Zia]

Hello..
the spider list in vbulletin.com now being updated...get it from there
http://www.vbulletin.com/forum/showpost.php?p=565415&postcount=12
Thanx to Stadler

:) also thnx to mikel..unless this watcher it wasnt possible to get all those spiders name :)

umm can u say..now most of the spider is now being catagorised...
after update the new spiderlist.....
unknown spider will changed to known spider?

can u have look on this lists catagori & update the watchers catagory stats..

plz give a serious thought abt sql query.
* Zia offer mikel an ice cold chilled bear

[Svenna]

hi mike,

here my german translation.

bye und best regards,
svenna

[kofoid]

mine isn't working either..... no spiders display

[DementedMindz]

the format that is used on vbulletin.com wont work with it... even the one zia post wont either... i posted a few pages back a working one... the format they have one vbulletin.com is wrong and its not even like that stock so i have no clue why they update the list there cause its still a wrong format

[TheBlackPoet]

i guess i dont know enough about spiders to ask the right questions.. but i do know enough to read threads to determine what is good or bad for my forum. and i guess some spiders are good for search engine purposes... while others milk your bandwidth....i installed this watcher, but i am now looking for a bad spider SNIPER.....
any ideas?

[Zia]

the format that is used on vbulletin.com wont work with it... even the one zia post wont either... i posted a few pages back a working one... the format they have one vbulletin.com is wrong and its not even like that stock so i have no clue why they update the list there cause its still a wrong format

DementedMindz..
its really a strange matter...it should not be wrong..its published by Stadler on vb.com.

And the latest one is working for me. the new spider is listed/update/entried there..they are showing over on my spiders.php page..

do u mind to drop few lines on vb.com ,in the thread (http://www.vbulletin.com/forum/newreply.php?do=newreply&p=1146107) regarding the error u found ?

I think :
let me xplain :
a site got a spider Google-feedfetcher - 100 times.

Google-Feedfetcher : wastnt listed on the old spiderlist.xml , so it was appeared as un-known spider's block.

after replace the new spiderlist.xml : if again google-feedfecther spider again crawl the site
visit no#101(100 times on old list.xml) , it will be appeard in rss-spider block.

The old data base will not be modified...
its my point of view.

Edit: plz see this hacks description mikelbeck refer the same thread of vb.com. So i dont think mikel use any other format.
To make sure that you can decode the maximum amount of spiders, you should grab the latest spiderlist.xml and replace the spiders_vbulletin.xml file in your forumhome/includes/xml/ directory with the one from this thread: http://www.vbulletin.com/forum/showthread.php?t=76662



Thnx.

[DementedMindz]

DementedMindz..
its really a strange matter...it should not be wrong..its published by Stadler on vb.com.

And the latest one is working for me. the new spider is listed/update/entried there..they are showing over on my spiders.php page..

take a look at stock vbulletin and take a look at stadlers its wrong... i have fixed it on mine but i dont use this no more as of the high number of queries on that page... alot of members here were saying it wasnt working for them either and then once i posted mine with how the format was with the stock vbulletin file it worked perfect for them... i havent looked at the file now but it was way off last time...

here is the top of stadlers

<?xml version="1.0" encoding="ISO-8859-1" ?>
<!DOCTYPE spiderlist (View Source for full doctype...)>
- <!-- $Id$
-->


here is how it should be just like it is on the stock vbulletin file


<?xml version="1.0" encoding="ISO-8859-1"?>

<searchspiders>

[mikelbeck]

Ok, people, I've had some time to get back to this. Sorry it's taken so long....

I've got Beta 10 running here: http://www.heavymetalscene.com/spiders.php

Notice how much faster it is. I've pulled out the "rolling up" code and placed it in a file that's run once an hour via a cron job. So the data isn't 100% up to date, but it uses much (much) less queries than the previous version.

What else do you want to see in this release?

[DementedMindz]

how many queries are on that page? also will it auto update once a new spider is added?

[mikelbeck]

how many queries are on that page?

8! :classic:

No matter how much data is in the database, it's still 8 queries.

http://www.heavymetalscene.com/spiders.php (not a whole lot of spider data): Page generated in 0.19605 seconds with 8 queries
http://www.forumowners.com/spiders.php (quite a bit of spider data): Page generated in 0.51553 seconds with 8 queries
http://www.listedwrong.com/forums/spiders.php (very little spider data): Page generated in 0.14583 seconds with 8 queries

also will it auto update once a new spider is added?

When a new spider is added to the xml file you mean? I'll see if I can make it so it'll evaluate the spider xml file and update the "unknown" spiders if there's new data.

[DementedMindz]

yes that is what i ment sorry... yeah that would be great and glad to see you brought them down...

[Zia]

its nice mikelbeck to knowing abt the new version..

please give thought about not lost old spider visitng data after install new version..
:) Less query is really delicious things

* Zia is waiting for the new release

[Stadler]

Can you please explain what you changed in my file?

btw: you shouldn't relay on the root-tag. Freddie implemented the xml-file before I've finished the DTD. However: online.php of vBulletin works with both root-tags, since it simply ignores it.

Regards,
Christian Stadler

[lsgworldl]

excellent

[4yBak]

why do you save all data without parsing in field PAGE?
Example:
faq, index, showthread, member, forumdisplay, calendar, memberlist, register, faq, showthread, showthread, showthread, showthread, showthread, index, showthread, showthread, showthread, showthread, showthread, index, member, showthread, login, member, sho
So, as you see, in this field has many repetitions like showthread, index, member, etc.

I think, in DB must be just one mention about each page.

PS: you can do that in this statistics was include information when spiders visited forums archived pages? Thanks.

[oberheimhaven]

Can someone throw me the code to add this to my navbar please I know Im learning thank for the great hack installed 3.5.4 no issues at all Just need quick code to add to nav bar

thxs
Mark

[oberheimhaven]

I got it I added to tool bar fav I dont need this inside my actually forums works great thanks

Markus

[Snake]

Ok, people, I've had some time to get back to this. Sorry it's taken so long....

I've got Beta 10 running here: http://www.heavymetalscene.com/spiders.php

Notice how much faster it is. I've pulled out the "rolling up" code and placed it in a file that's run once an hour via a cron job. So the data isn't 100% up to date, but it uses much (much) less queries than the previous version.

What else do you want to see in this release?
Oh that is great but have you updated the zip pack yet?

[mikelbeck]

Oh that is great but have you updated the zip pack yet?
Not yet, give me a couple of days.

[Snake]

Alrighty then, I'll be waiting here for the update.

[jd_sa_bowler]

Hi guys,

like the mod, I have installed by I am unsure if it is working correctly.

mysite http://www.bowls-aust.com/spiders.php

The problem is no type display for spiders except unknown and the google spider seems to be in the correct address range to be identified by the spiders_vbulletin.xml file data.

I uploaded the latest spider file and rename to spiders_vbulletin.xml

Am I just to impatient or is something wrong.

[Stadler]

I don't know, if this is already suggested, but I suggest, that you add an option to only list spiders with at least X Hits in the list of unknown spiders and perhaps add a link to 'List all unknown spiders' then. especially these garbled user agents, like 'cblbshjhbjdmpvrbfdrvv5arh5wi' for example seem to me like some crappy configured firewall or proxy, which tries to hide the user-agent ...

[Svenna]

I straight one mySQL errors indicated got. Where does the problem lie?

Invalid SQL:
DELETE FROM spider_watcher WHERE bot = 'Netluchs/0.8-dev ( ; http://www.netluchs.de/; ___don't___spam_me_@netluchs.de)';

MySQL-Error : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 't___spam_me_@netluchs.de)'' at line 1


Any can help?

[4yBak]

Any can help?
what you do when this error going? Query like this may be possible SQL-injection did posible hackers, but I'm not sure about it, so don't worry early

mikelbeck
can you do display IP of spiders via popupable div? Because JScript popup don't usable :(

This plugin don't want add in list spider WebAlta. In table spider_watcher I see always random fileds like this:
56680 | WebAlta Crawler/1.3.15 (http://www.webalta.ru/bot.html) | Webalta Crawler | 2006-07-14 20:18:55 | 1373540995 | showthread | searchspider | http://www.webalta.ru/bot.html | 0
56681 | WebAlta Crawler/1.3.15 (http://www.webalta.ru/bot.html) | Webalta Crawler | 2006-07-14 20:19:42 | 1373540995 | showthread | searchspider | http://www.webalta.ru/bot.html | 0
...

So, as you see - script do not count rollup. Why?

[Snake]

No update yet?

[Nathan2006]

Thank you mikelbeck

Really nice :D

Install

[Nathan2006]

To make sure that you can decode the maximum amount of spiders, you should grab the latest spiderlist.xml and replace the spiders_vbulletin.xml file in your forumhome/includes/xml/ directory with the one from this thread: http://www.vbulletin.com/forum/showthread.php?t=76662

How do I update the spider list?

I had a look on there but I dont know how to

On this post its from 2001
http://www.vbulletin.com/forum/showpost.php?p=565415

Thank you for any help :)

[Zia]

helo mikelbeck..

is there any update gonna release for vb3.6.0 ?

its an important module.

thanks

[4yBak]

is there any update gonna release for vb3.6.0 ?

its an important module.
this product compatible with vB 3.6 Beta 2

[ILTK]

Quick question:

I see the numbers going up for example on the googlebot entry, but the date is not allways updated.

Is this because you only update the date when it's a GET and not a HEAD request? or could this be a bug?

Love the hack btw.

[big dan]

Great plugin, installed without a hitch and works perfectly for me on 3.5.4

Thanks! :)

[libertylounge]

On my forum, it's not recognizing what bots are what.. how can I remedy this?

They are currently all listed as "Unknown Spider"

edit:

Ok, I downloaded that spiderlist.xml from vbulletin.com and replaced my spiders_vbulletin.xml with it.. is there a way to "update" the current entries so I'm able to see which bots are which easier, or will that never update?

[Aneurysm]

Thanks! easy install, works great.

[ILTK]

Had some new spider break the script today, looks like some of the code in the user_agent made the sql bomb, I have removed the script until this can be resolved, this isn't affecting my forums, just when you load the spiders page.


Database error in vBulletin 3.5.4:

Invalid SQL:
DELETE FROM spider_watcher WHERE bot = '&lt;a href='http://www.netforex.org'&gt; Forex Trading Network Organization &lt;/a&gt; info@netforex.org';

MySQL Error : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'http://www.netforex.org'&gt; Forex Trading Network Organization &lt;/a&gt; info@' at line 1
Error Number : 1064
Date : Saturday, August 5th 2006 @ 11:43:32 AM
Script : http://www.<snipped by iltk>
Referrer :
IP Address : 83.*.*.*
Username : <snip>
Classname : vb_database

[mikelbeck]

I realized it's been a ridicously long time since I've updated this mod, but I did it today...

This version dramatically cuts down on the number of SQL queries that occur when somebody views the spiders page. Instead of rolling up the spider data at that point, a cron job rolls up the data once an hour.

Please note that the installation and upgrade instructions have changed, and a new file is included.

ILTK - the error you've just posted should be corrected in this latest version.

I don't have 3.6 installed on any of my sites yet, so I can't confirm it works with it. I don't see why it wouldn't, but I can't say that it does for sure.

[jw00dy]

This is cool *installed*

Does this new version take some time before it shows any spiders? I'm guessing it does. I'm going to see if I can kick some spiders over to check my forums and gen some stats.

Thanks

[ILTK]

Sweet, thanks for the update, my hostings cpanel is dog slow so I used it alot instead of going into awstats to see spidering trends.

[mikelbeck]

This is cool *installed*

Does this new version take some time before it shows any spiders? I'm guessing it does. I'm going to see if I can kick some spiders over to check my forums and gen some stats.

Thanks
Yes, the rollup runs once an hour, so it'll take that time until you see some stats. I looked at your site, I see Google and MSN on the spiders page. ;-)

[Mudvayne]

mikelbeck.. Any plan to release it for vB 3.6 ?

[mikelbeck]

mikelbeck.. Any plan to release it for vB 3.6 ?

Once I can upgrade my sites to 3.6 and test it, yes.

Does it not work with 3.6?

[jw00dy]

Yes, the rollup runs once an hour, so it'll take that time until you see some stats. I looked at your site, I see Google and MSN on the spiders page. ;-)
Sure enough, thank you. I love it. I'm always wondering if I'm getting index'd. :up: and by who...

[Hornstar]

Okay I will know in an hour if it is working fine with Vbulletin 3.6 Gold. The installation went fine, The instructions was a bit confusing for the cron job, hopefully I have set that up correctly for the hourly run of it. (Maybe that could be re-worded to be more noob friendly for people like myself ^^ lol)

[blogtorank]

Okay I will know in an hour if it is working fine with Vbulletin 3.6 Gold. The installation went fine, The instructions was a bit confusing for the cron job, hopefully I have set that up correctly for the hourly run of it. (Maybe that could be re-worded to be more noob friendly for people like myself ^^ lol)

Hope this helps you or anyone else that needs help in the cron per his instructions, here is a visual for you inside a cPanel CRON Job area see the attachment below ;)

[Nathan2006]

***UPGRADE NOTE***
When you upgrade from version 1.0.0 Beta 7 to 1.0.0 Beta 8 your existing spider data will be lost!


I have just upgraded it and all the spider counts are still there :D

Working great! now 8 queries less and very pleased with this

Thank you :)

[mikelbeck]

I have just upgraded it and all the spider counts are still there :D

Right, it said when you upgrade from Beta 7 to Beta 8, you just upgraded to Beta 10 so this line doesn't apply.




Working great! now 8 queries less and very pleased with this

Thank you :)
Good to hear!

[ILTK]

Something I been wondering, why is it that the hits seem to update allways, but the last visit date doesn't allways update?

[mikelbeck]

Something I been wondering, why is it that the hits seem to update allways, but the last visit date doesn't allways update?

I'll look into that, I think it has something to do with how the data is selected during the rollup. The latest timestamped record may not be the last record selected, so the timestamp is wrong. I think, I'm not positive...

[jw00dy]

Not sure if anyone has said this yet, but this seems to be working fine on vB 3.6.0 Gold :)

[mikelbeck]

Not sure if anyone has said this yet, but this seems to be working fine on vB 3.6.0 Gold :)

Good to hear.

[ninjashoes]

Something I been wondering, why is it that the hits seem to update allways, but the last visit date doesn't allways update?
Since I upgraded my last visits arent updating at all.

[ninjashoes]

It seems as though the task dissapeared, I readded it but am not sure what to put for the Varname section. My last visit time still isnt updating, when its a new spider then it shows up but for google the last time it shows was before I upgraded to 3.6

[blind-eddie]

Add a link to spiders.php on your navbar or footer.???
If this is correct:http://outerzone.us/forum/includes/spiders.php or http://www.outerzone.us/spiders.php I'm getting a 404 error
Yes I'm a noob....Any help would be greatly appreciated.

[Neal-UK]

Add a link to spiders.php on your navbar or footer.???
If this is correct:http://outerzone.us/forum/includes/spiders.php or http://www.outerzone.us/spiders.php I'm getting a 404 error
Yes I'm a noob....Any help would be greatly appreciated.

http://www.outerzone.us/forum/spiders.php

[blind-eddie]

Amesome Thank You for helping me out

[blind-eddie]

To the people who have installed this - can you post a link to your spiders page?

http://outerzone.us/forum/spiders.php

[blind-eddie]

Ok one last request for this great hack,I can add a link to This other Great Hack (https://vborg.vbsupport.ru/showthread.php?t=104725&highlight=menucontrol) ,but I would really like to add this to my 1st or 2nd navbar but I'm not exactly sure how to. I do know how to get to navbar, I see items that are in the navbar,but I dont know how to edit an existing item with proper wording to get The Spider Page to show up on 1st or 2nd navbar.......

[jw00dy]

Go to -> vBulletin Options -> Menu Control Version 1.3 Final Settings

Then add your link.

Name of menu Item: spiders
Menu link one: outerzone.us/forums/spiders.php

That's it. customize anything else you want to use.

[jw00dy]

Actually, I just looked and it looks like you figured it out.

[doopz]

I'm wondering..

Database error in vBulletin 3.5.4:

Invalid SQL:
DELETE FROM spider_watcher WHERE bot = '';show processlist;';

MySQL Error : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ';show processlist;'' at line 1
Error Number : 1064

What exactly is this?
I don't see 'show processlist' anywhere in the php script, making me wonder it might be someone trying to exploit the script?

havnt look into it more but..

edit:
I see some entries in the mysqldatabase in the table 'bot'.

51219 ';show processlist; Unknown Spider 2006-08-16 22:33:48 1113767976 index 0
51220 ';show processlist; Unknown Spider 2006-08-16 22:33:52 1113767976 index 0
51223 ';show processlist; Unknown Spider 2006-08-16 22:34:08 1113767976 index 0
51399 ';show processlist; Unknown Spider 2006-08-16 22:54:18 1113767976 index 0
51402 ';show processlist; Unknown Spider 2006-08-16 22:54:22 1113767976 index 0
51403 ';show processlist; Unknown Spider 2006-08-16 22:54:27 1113767976 index 0
51404 ';show processlist; Unknown Spider 2006-08-16 22:54:31 1113767976 index 0
51405 ';show processlist; Unknown Spider 2006-08-16 22:54:35 1113767976 index 0
51406 ';show processlist; Unknown Spider 2006-08-16 22:54:38 1113767976 index 0
51407 ';show processlist; Unknown Spider 2006-08-16 22:54:43 1113767976 index 0
51408 ';show processlist; Unknown Spider 2006-08-16 22:54:47 1113767976 index 0
51409 ';show processlist; Unknown Spider 2006-08-16 22:54:51 1113767976 index 0

and some more nasty entry's

[Hornstar]

I have the latest one installed, and ive not had any problems with it and had it for about a week. Then I had to restart my sever today and then got this database error over and over and over hundreds of times.


Database error in vBulletin 3.6.0:

Invalid SQL:
INSERT INTO vb3_spider_watcher (nice_bot, bot, ip_address, page, type, info, timestamp) VALUES ('Unknown Spider','Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.6) Gecko/20060728 Firefox/1.5.0.6','1081014013','showthread','','', NOW());

MySQL Error : Duplicate entry '4504085' for key 1
Error Number : 1062



Any ideas why?

[4yBak]

Some records from MySql Dump:
(118075, 'WebAlta Crawler/1.3.18 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)', 'Webalta Crawler', '2006-08-23 20:02:30', '1373540995', 'member', 'searchspider', 'http://www.webalta.ru/bot.html', 0),
(118076, 'WebAlta Crawler/1.3.18 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)', 'Webalta Crawler', '2006-08-23 20:03:13', '1373540995', 'member', 'searchspider', 'http://www.webalta.ru/bot.html', 0),
(118077, 'WebAlta Crawler/1.3.18 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)', 'Webalta Crawler', '2006-08-23 20:03:56', '1373540995', 'member', 'searchspider', 'http://www.webalta.ru/bot.html', 0),

My board visit crawler WebAlta (info for this crawler exist in my spiders_vbulletin.xml), in db records for this crawler exist, but this hack don't count visits of this crawler (field rolup already = 0). With other spiders I don't have this problem, only with WebAlta. Why ?

PS: I'm using version of this hack Beta 10, but when I use version beta 9 - this error still happened.

Need help!

[mikelbeck]

It's because that spider is identifying itself as "Windows NT", so the rollup process thinks it's a user and not a spider. I guess it should check to see if there's anything in the "type" field before deleting it...

I'll fix that in the next version.

[Snake]

Good to see it's working fine on 3.6. I'm on it! :D

[4yBak]

thx, waiting for the next version :)

[Snake]

So when is it gonna be released? Anytime soon? :)

[Snake]

Also it's asking me for the varname while I was adding the task in the acp.

[mikelbeck]

Yep, the varname thing is new for 3.6, I'll have to update the docs for the next version.

As for when the next version will be out, I can't say. I haven't had a chance to do anything with it, and I won't be able to get to doing anything until next week at the earliest.

[Hornstar]

I just tried to do a backup of my database through SSH and got this error for the first time


mysqldump: Got error: 1016: Can't open file: 'vb3_spider_watcher.MYI' (errno: 145) when using LOCK TABLES



any idea why?

[jw00dy]

I have no idea, but you could maybe try disabling the plug-in, then backing it up.

Or maybe it just happened to be running the cron when it tried to backup that table?

Just some ideas to try.

[mac27]

I installed this great hack. I have also updated my spider list. I had a spider crawl my forum and in the spider.php page it named it unknown spider. Anyone have an idea. It is the gigabot spider.

The gigabot spider is listed in my spider listing also.

[4yBak]

I'm still waiting fix for this bug - https://vborg.vbsupport.ru/showpost.php?p=1059425&postcount=396 ;)

[Zia]

Yep, the varname thing is new for 3.6, I'll have to update the docs for the next version.

As for when the next version will be out, I can't say. I haven't had a chance to do anything with it, and I won't be able to get to doing anything until next week at the earliest.

hi mike...
when the next version going to release? also for 3.6.1?

[Snake]

Any updates? We have been waiting for weeks now.

[rezapci]

Installation Instructions
1. Upload spiders.php to the root of your forum.
2. Upload spiders_rollup.php to the includes/cron directory.
3. Import the file product-spiderwatcher.xml using the Manage Products module.
4. Add a link to spiders.php on your navbar or footer.
5. Add a cron job with the following information:
Title: Spider Watcher Rollup
Day of the Week: *
Day of the Month: *
Hour: *
Minute: 0 - - -
Log entries: Yes
Filename: ./includes/cron/spiders_rollup.php

where in navbar? with what code?

how I add the cron job?

your Instructions is not clear...!

[Snake]

What he meant is to use the following code to your navbar template, which is optional.

<td class="vbmenu_control"><a href="spiders.php$session[sessionurl_q]">Spiders</a></td>

[tracylee]

I tried uninstalling this for a client as we're slimming down for a server move.
I got a call that guests were unable to view the forum. I logged out and got a database error. Followed earlier advice to view source on the page and found that it was still trying to insert my information as a guest into the now gone "bots" table.

I read through the xml file and found that the hook is global_start, so I looked at the global.php file, and sure enough, the SQL query to insert data is still in there.

I'm not sure what about uninstalling should have turned that off? The plugins were gone, the templates were gone - I had to create the table again just to finish researching it.

Anyway, just thought that this might be helpful if anyone else hits a problem uninstalling.

[Zia]

any more update news ?

[Snake]

Yeah we have been waiting for months now...

[Zia]

Knock Knock Mikel Beck
Ur at home???

[4yBak]

I'm still waiting fix for this bug - https://vborg.vbsupport.ru/showpost.php?p=1059425&postcount=396 ;)

already 2 month goes and nothing news from author of this hack. Are you will be support this hack or not?

Also I have few suggestions:
- create an option or to do it by default that column with Unknown spiders can see only admin. It's need because many user replace in his browsers User-Agent for obscene language etc. and all this information will index search spiders
- add checkbox opposite each row and add button Remove selected rows. So... we can remove some records that not needed just from spiders.php file. Sure, this checkbox must be displayed only for specifed usergroup (for example administrator). It will be very comfortable ;)

[Zia]

working with vb3.6.4 ?

any news on reduce SQL load ?

[Zia]

any news for 3.6.4 ?

[Snake]

Yep. :)

[4yBak]

I don't know why in hack info checked value Supported if support is ... :(

This hack very good, but without support it not needed :(

[Ohiosweetheart]

This works perfectly on my 3.6.4 board

[mikelbeck]

Author guy here...

I really haven't had the chance to think about doing anything with this, let alone actually doing anything. And I'm not sure when I will have. I've kind of gotten away from the vBulletin stuff lately, I've sold off all of my vBulletin sites and don't see me having any more in the near future. I just don't have the time anymore...

If somebody else wants to pick this up and run with it, go for it!

[Zia]

a bad news indeed mikel...

its an imp things..

umm any one out there?

[Snake]

Yeah is there anyone to take over this mod? :(

That's so sad, mikel.

[mikelbeck]

Yeah is there anyone to take over this mod? :(

That's so sad, mikel.

Sorry... I got a new job that takes up most of my time, and I haven't had time to even back to the boards that I'm a "regular" on. So I unloaded all of my own sites and kept the ones that need no maintenance.

[Trana]

Ack, check this bad boy out:

Database error in vBulletin 3.6.4:

Invalid SQL:
INSERT INTO spider_watcher (nice_bot, bot, ip_address, page, type, info, timestamp) VALUES ('Unknown Spider','HouxouCrawler/Nutch-0.8.2-dev (houxou.com's nutch-based crawler which serves special interest on-line communities; http://www.houxou.com/crawler; crawler at houxou dot com)','3276309319','showthread','','', NOW());

MySQL Error : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 's nutch-based crawler which serves special interest on-line communities; http://' at line 1 Error Number : 1064
Date : Wednesday, January 3rd 2007 @ 08:04:43 PM

I think we need some field validation here.

[behcet]

hello friends,
i have a 500 mb database. my spider watcher table is enlarging day by day. and at this moment it is using 350 mb! why does it enlarge? is it normal situation? how can i stop it? or if i uninstall the spider may i meet any problems? what will you suggest me? tnkx alot...

[jw00dy]

I unistall mine periodically just for this same reason. Just uninstall, then re-import and it will start over. You will lose all previous data, so be warned.

[Smitty]

Has anyone tried this on 3.7.2?

[Jaxel]

Does this work on 3.7.2?

[4yBak]

work on 3.8.1 pl1 (but it was previously installed at vb3.5.x or vb3.6.x)