Spider Watcher

[mikelbeck]

Spider Watcher
Author: Mikel Beck (mikel.beck@elite-computing.net)

This hack keeps track of the spiders (Search Engine robots) that visit your fourm. Every time a guest visits a page, the guest's IP address, user agent and the page they visited are logged to the database.

When somebody views the spider statistics page, this data is "rolled up", meaning the raw data is collated, the spider's name is determined by comparing the user agent to data contained in the spiders_bulletin.xml file, and the number of pages and visits is summarized and writted back to the database. In addition, and data from non-bots is removed.

The data is then displayed in a easy to read format for your viewing pleasure.

If the user viewing the report has permissions to view IP addresses, these are displayed as well.

A live version of the report from one of my sites can be seen here: http://www.happyhourpub.com/spiders.php

Also see the attached screenshot for an exmaple.


Revision History:
1.0.0 Beta 1 - 01/05/2006
- Initial Release

1.0.0 Beta 2 - 01/06/2006
- Included templates for spiders.php
- Removed text from templates, added them as phrases

1.0.0 Beta 3 - 01/07/2006
- Split up the display of "known" and "unknown" spiders

1.0.0 Beta 4 - 01/25/2006
- Corrected potentional SQL injection issue in plug-in
- Reduced the number of SQL queries required to display statistics
- Corrected date/time display issue

1.0.0 Beta 5 - 02/01/2006
- Reduced the number of SQL queries required to display statistics

1.0.0 Beta 6 - 02/08/2006
- No release

1.0.0 Beta 7 - 02/11/2006
- Corrected issue with "unknown" spiders not being displayed properly.
- Added tracking of the type of spider (searchspider, link checker, etc)

1.0.0 Beta 8 - 02/19/2006
- Change the display of IP addresses to be a pop-up so they're all not displayed on the main page.
- Combined the spiders that have the same name but different user agents.

1.0.0 Beta 9 - 03/10/2006
- Changed the display to group similar spiders together (search spiders, http check spiders, etc)

1.0.0 Beta 10 - 08/08/2006
- Changed how the rollup functions. Instead of rolling up every time somebody views the spider page, it rolls up once per hour.
- Corrected a few bugs here and there, mostly related to removing entries from the database.

Installation Instructions
1. Upload spiders.php to the root of your forum.
2. Upload spiders_rollup.php to the includes/cron directory.
3. Import the file product-spiderwatcher.xml using the Manage Products module.
4. Add a link to spiders.php on your navbar or footer.
5. Add a cron job with the following information:
Title: Spider Watcher Rollup
Day of the Week: *
Day of the Month: *
Hour: *
Minute: 0 - - -
Log entries: Yes
Filename: ./includes/cron/spiders_rollup.php

Upgrade Instructions
1. Upload (and overwrite) spiders to the root of your forum.
2. Upload spiders_rollup.php to the includes/cron directory.
3. Import the file product-spiderwatcher.xml using the Manage Products module. Make sure the "Allow Overwrite" option is set to "Yes".
4. Add a link to spiders.php on your navbar or footer.
5. Add a cron job with the following information:
Title: Spider Watcher Rollup
Day of the Week: *
Day of the Month: *
Hour: *
Minute: 0 - - -
Log entries: Yes
Filename: ./includes/cron/spiders_rollup.php

***UPGRADE NOTE***
When you upgrade from version 1.0.0 Beta 7 to 1.0.0 Beta 8 your existing spider data will be lost!

To make sure that you can decode the maximum amount of spiders, you should grab the latest spiderlist.xml and replace the spiders_vbulletin.xml file in your forumhome/includes/xml/ directory with the one from this thread: http://www.vbulletin.com/forum/showthread.php?t=76662

[The Chief]

nice, I'll check it out later on :)

[eViL_sTiGmA]

How do i see spiders who visited my site????

just go to www.mysite.com/forum/spiders.php

??

[swa]

doesn't work on 3.51

[mikelbeck]

I've just realized that the required templates aren't included in the zip file.

Sorry about that, I'll correct this in the morning and will post an updated archive.

[Zia]

Its Nice...really nice...been looking for it...for a long time....but what about the spider list? did that detect allmost all spider or only common few like yahoo,google,msn...
els this there are many spiders are out there....hope it is able to detect all of spider...

Thanx

[mikelbeck]

Its Nice...really nice...been looking for it...for a long time....but what about the spider list? did that detect allmost all spider or only common few like yahoo,google,msn...
els this there are many spiders are out there....hope it is able to detect all of spider...

Thanx
It will detect any of the spiders listed in this file:

To make sure that you can decode the maximum amount of spiders, you should grab the latest spiderlist.xml and replace the spiders_vbulletin.xml file in your forumhome/includes/xml/ directory with the one from this thread: http://www.vbulletin.com/forum/showthread.php?t=76662

The way it sits now it's not working, I'm having an issue with the templates for the spiders.php page. I'll have a functional version in the morning. Sorry for this.

[mikelbeck]

The zip file for this hack has been updated to include the templates for spiders.php. If you have already installed this, you can re-install it, just be sure to check the "Allow Overwite" is checked when you upload the xml.

[mikelbeck]

Beta 2 of this modification is available.

- Included templates for spiders.php
- Removed text from templates, added them as phrases

Upgrade Instructions
1. Uninstall the previous version using the Manage Products module.
2. Upload (and overwrite) spiders to the root of your forum.
3. Import the file product-spiderwatcher.xml using the Manage Products module.
4. Add a link to spiders.php on your navbar or footer.

To the people who have installed this - can you post a link to your spiders page?

[nexialys]

2 thumbs on the spiders... :)

[eViL_sTiGmA]

Now works, thank you, i love it =D

[Snake]

Whoa just installed this hack and works great! Thanks bro!

[mikelbeck]

Whoa just installed this hack and works great! Thanks bro!

Nice to hear that!

Do you have a link to where you have it installed?

[puertoblack2003]

doesn't work on 3.51


it do work got it installed. :ermm:

[silurius]

I don't know why, but I love this hack!

*Clicks install

Could you briefly outline the justification for watching spiders? Is there a real business or security benefit, or is it just satisfying?

Also, I assume there is no security risk associated with allowing any old viewer to view the Spider Watcher page?

[Allan]

Thanks ;)

Click Install

[Hornstar]

I'm going to install this as i requested this a while back. This is what i've been looking for ^^

nice work. Hope to see more from you :D

[bashy]

Hi

Thanks for this, I have installed,
Working very well thanks :)
Heres the link for the working hack ;)

http://www.bashys-place.com/forums/spiders.php

Regards

Bashy

[Lizard King]

Working pretty good thanks

i have the latest vbulletin_spiders.xml but i still get unknown spiders.
Check the list http://www.ayyas.com/spiders.php

[mikelbeck]

I don't know why, but I love this hack!

*Clicks install

Could you briefly outline the justification for watching spiders? Is there a real business or security benefit, or is it just satisfying?

Also, I assume there is no security risk associated with allowing any old viewer to view the Spider Watcher page?

I think it's just satisfiying, to know that your hard work on the SEO for your forum attracts spiders... And with this you can see when their last visit was, and how many pages they've viewed.

As for being a security risk, I don't see how it would be.

[Zia]

Dear mikelbeck

Thankx..for this too kool hack...but when u r gonna relase final version...

and one more point....if guest access is disable in forum..then can spider crawl the forum?
If Not..Any way that can makes spider crwal the forum ????
It changes db...hope it wont create any probs in sql db..

:) ALso Looking For More Modified hack...

Thankx...waiting for final version ....



If Not...can u make that the spider can crawl forum..if guest access disable.

[mikelbeck]

Dear mikelbeck

Thankx..for this too kool hack...but when u r gonna relase final version...

I've got one or two things I'd like to add/change on this. Once that's done, I'll leave it in beta for a little while to make sure there's no issues, and once that's done then I'll release the 1.0.0 version.

and one more point....if guest access is disable in forum..then can spider crawl the forum?

No, I'm pretty sure you need to allow guest access to your forums for the spiders to get in. The spiders don't log in with a vBulletin username, so they are guests.

If Not..Any way that can makes spider crwal the forum ????

Not that I'm aware of. Unless you enable the archive and allow guests to view that, I guess.

It changes db...hope it wont create any probs in sql db..

It doesn't.

:) ALso Looking For More Modified hack...

What else would you want to see in it?

If Not...can u make that the spider can crawl forum..if guest access disable.

That's got nothing to do with this hack, that has to do with how you configure your forum.

[bashy]

Hi

Is there anyway to make a Unkown spider know?
I have the upto date spider xml, so not sure why it would be like this?

Bashy

[mikelbeck]

Hi

Is there anyway to make a Unkown spider know?
I have the upto date spider xml, so not sure why it would be like this?

Bashy
No, it's only able to decode those that are listed in the updated xml file. You could update your xml file, or ask the people who are maintaining it to add the spiders that you're seeing.

[bashy]

ok no worries thanks anyway, great hack :)

Bashy

No, it's only able to decode those that are listed in the updated xml file. You could update your xml file, or ask the people who are maintaining it to add the spiders that you're seeing.

[LBSources]

works well .. thanks!

/me clicks install.

[nitro]

Looks like its working ok in 3.5.0 patched :)

Any easy way to add the whats going on or more importantly the who's online to the page ?

[mikelbeck]

Looks like its working ok in 3.5.0 patched :)

Any easy way to add the whats going on or more importantly the who's online to the page ?

Do you mean the box that's display at the bottom of forumhome?

I would think that you could just copy that block from the forumhome template and put it into the spiderwatcher_spiders template.

[bashy]

Hi

I am also interested in this, could you please tell me exactly the right code to copy from forumhome and to exactly where to put it please?

Regards

Bashy


Do you mean the box that's display at the bottom of forumhome?

I would think that you could just copy that block from the forumhome template and put it into the spiderwatcher_spiders template.

[nitro]

Do you mean the box that's display at the bottom of forumhome?

I would think that you could just copy that block from the forumhome template and put it into the spiderwatcher_spiders template.

That would just display the block and images with no info. Guess I will have to try an find the php code in vb files and take pot luck on adding it to the spiders.php

[Lizard King]

That would just display the block and images with no info. Guess I will have to try an find the php code in vb files and take pot luck on adding it to the spiders.php

https://vborg.vbsupport.ru/showthread.php?t=104591

[nitro]

Thanks that works great best click install there aswell ;)


Iv noticed this spider product also detects JumpD's RSS article bot as an unknown spider.

I dont mind that as it shows that bots hit counter aswell but how would I add to the spiders xml file this bot which has an ip of 0.0.0.0 and i dont think it has an agent but I maybe wrong so it shows as say Articlebot instead of unknown ?

[Zenith]

Tidy hack, thanks.

I notice that the last visited date is out though. I only installed a few minutes ago but the few spiders already listed are showing "Yesterday". Time is correct, but it should be "Today".

Clicked install...

[mikelbeck]

Version 1.0.0 Beta 3 is available.

1.0.0 Beta 3 - 01/07/2006
- Split up the display of "known" and "unknown" spiders

Upgrade Instructions
1. Upload (and overwrite) spiders.php to the root of your forum.
2. Import the file product-spiderwatcher.xml using the Manage Products module. Make sure the "Allow Overwrite" option is set to "Yes".
3. Add a link to spiders.php on your navbar or footer.

[Zia]

Thankx Man.. for real kool support..


-Unknown Spider
Googlebot-Image/1.0 01-07-2006
01:09 AM 66.249.65.202 errordocs 2

-Unknown Spider
Mozilla/3.01 (compatible;) 01-07-2006
03:48 AM 69.88.30.146, 202.52.197.55, 202.52.197.56, 210.50.143.21 cron, errordocs 213

-Unknown Spider
Mozilla/4.0 (compatible; Google Desktop) 01-07-2006
11:55 AM 70.236.181.179 errordocs, external 7



umm one spider hit 213 times..its detected as unknown spider..
can u plz mark them?? and most of time it face errordoc..
how could we detect that..right where it face error doc?
we need to know which bot see error doc in where?


and which bot is doing harm to forum...any way to detect from the tracker page?

plz thanks...help us

[bashy]

Hi..

What i find intersting is the fact that all the spiders are showing as being online 20 hours ago...I know for a fact that most have been in all day long, they very rarely leave ;) so how can it be saying 20hrs ago or should i say why is it saying 20 when it clearly isnt lol

Please advise?

Bashy

Spider Name (Known Spiders) Last Visit IP Address(es) Page Hits
Google AdSense 20 Hours Ago
08:37 PM 66.249.66.102 THIS_SCRIPT, forumdisplay, index, login, member, misc, newreply, newthread, online, pager, poll, private, profile, register, search, showthread 347
Google 20 Hours Ago
08:48 PM 66.249.66.102 index, showthread 25
MSNBot 20 Hours Ago
04:46 PM 207.46.98.48 VBGooglemap, calendar, forumdisplay, index, jukebox, member, memberlist, search, showthread, vbradio 44
Yahoo! Slurp 20 Hours Ago
08:10 PM 68.142.249.14, 68.142.249.160, 68.142.250.17, 68.142.251.166 forumdisplay, member, showthread 4

Spider Name (Unknown Spiders) Last Visit IP Address(es) Page Hits
Unknown Spider
20 Hours Ago
08:00 PM 82.110.105.18 external 76
Unknown Spider
FeedBurner/1.0 (http://www.FeedBurner.com) 20 Hours Ago
08:16 PM 66.150.96.109 external 30
Unknown Spider
Jakarta Commons-HttpClient/3.0-rc2 20 Hours Ago
08:16 PM 206.188.0.11, 209.18.119.137, 209.18.119.138 showthread 18
Unknown Spider
Mozilla/4.0 (compatible; Google Desktop) 20 Hours Ago
03:38 PM 86.142.103.72 external 5
Unknown Spider
Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/ysearch/crawling/crawling-02.html ) 20 Hours Ago
08:24 PM 209.191.83.13 showthread 47

[mikelbeck]

That is strange...

The only thing I can think is that the server time zone is set to something different from what yours is.

What the plugin does is write a record into the database, and the timestamp it uses is "NOW()", which is the current time. I think it uses the server's current time, but I'm not positive. So if the server's time is different from your time (if it's set to GMT, for example, and your time zone is GMT-4), then the times will be different.

I'll have a look to see if it can be set to the board's current time instead of using the "NOW()" function.

[bashy]

Hi mikelbeck

Cheers for the quick response...
The server is in the UK as i am therefore we are on the same timezone :)

Bashy

That is strange...

The only thing I can think is that the server time zone is set to something different from what yours is.

What the plugin does is write a record into the database, and the timestamp it uses is "NOW()", which is the current time. I think it uses the server's current time, but I'm not positive. So if the server's time is different from your time (if it's set to GMT, for example, and your time zone is GMT-4), then the times will be different.

I'll have a look to see if it can be set to the board's current time instead of using the "NOW()" function.

[mikelbeck]

Hi mikelbeck

Cheers for the quick response...
The server is in the UK as i am therefore we are on the same timezone :)

Bashy
Can you go into your admin control panel, click on "Maintenenace", then "Diagnostics", then select "MySQL Variables" in the "System Information" box and click "Submit". What value does it show for "timezone", and what is your timezone? And also, what is your forum's timezone set to?

[bashy]

Hi m8

MySQL Variables
system_time_zone GMT

My time zone is GMT
Forum Timezone is GMT also

I dont suppose this helps you any now does it lol

Bashy

[mikelbeck]

Hi m8

MySQL Variables
system_time_zone GMT

My time zone is GMT
Forum Timezone is GMT also

I dont suppose this helps you any now does it lol

Bashy

Not even a little bit!

I'm clueless as to why those dates are wrong, I'll have to look into it and will come up with a fix.

[bashy]

Didnt think it would help you with them all being GMT ;)
Will wait for a fix, no rush as its still recording them regardless :)

Regards

Bashy


Not even a little bit!

I'm clueless as to why those dates are wrong, I'll have to look into it and will come up with a fix.

[mikelbeck]

Can run an SQL query? I'm interested to see the results of this:

SELECT now() , curdate() , curtime();

Can you run that and post the results? Also please post what the current time in your area is.

[Zia]

hello mikelbeck

need help....can u plz check the link

http://www.golpo.net/forum/spiders.php

There are some unknown-spiders & they faced ErrorDocs..we need to know in which location spider faced the error docs...

and we need to know more about spider list..mean updated spider list..
can u tell me where to submit the spiders details plz??

[nitro]

You would need to add unknown spiders to the xml file in an editor afaik

to stop spiders roaming unauthorised areas you would if I have it right want to place a robots.txt file in your webroot with details on what pages you dont want them to view. It can take sometime for the robots.txt file to get cached by the spiders.

[Zia]

You would need to add unknown spiders to the xml file in an editor afaik

to stop spiders roaming unauthorised areas you would if I have it right want to place a robots.txt file in your webroot with details on what pages you dont want them to view. It can take sometime for the robots.txt file to get cached by the spiders.

Can u help me a bit? how could i stop robot to visit inside a thread.
just dont want spider face any error docs?

how could I do it??

[bashy]

hi mikelbeck

Just a little more on this time issue, i thought i'd mention that although the 20 Hours Ago for example is incorrect the 08:08 PM actually is correct, strange 'eh lol

Regards

Bashy

[mikelbeck]

hi mikelbeck

Just a little more on this time issue, i thought i'd mention that although the 20 Hours Ago for example is incorrect the 08:08 PM actually is correct, strange 'eh lol

Regards

Bashy

Did you do this yet? https://vborg.vbsupport.ru/showpost.php?p=868338&postcount=43

[DementedMindz]

mikelbeck i like your site i was wondering is there somewhere where we can get the Drinks Menu that you have on your site ?

[Devil Woman]

I was wondering if there is anyway you can prevent members from being able to see the spider page at all and prevent members from seeing that admin and mods are in the spider page on the who is online page

Thanks

[samu2]

When does it start showing the spiders? I have got 6 spiders on now but it is showing that I have none.

http://www.mumsonthenet.co.uk/forum/spiders.php

Thanks for the hack

[mikelbeck]

DementedMindz: That's a database that I bought and a custom interface that I wrote... For now, it's not available.

Devil Woman: Not right now.

samu2: It should start showing spiders right away.

[samu2]

Darn,it hasn't shown one yet :(

[mikelbeck]

Darn,it hasn't shown one yet :(
That's odd, it should pick them up right away. Did you get any errors when you installed it?

Could you try un-installing and re-installing?

[Devil Woman]

Will this ever be added to this hack maybe or will you ever consider adding this in at all?

Thanks

[samu2]

No erros.Just un installed and re installed.No change.

[mikelbeck]

Devil Woman: I was thinking about adding permissions, so you can set who can and who can't see the page. I guess I'll do it.

samu2: Can you verify that you have a plugin called "Spider Watcher" that's hooked into "global_start"?

[samu2]

Yeah,I do :)

[Devil Woman]

Thanks for that, is there anyway you can password protect the page for now at all to prevent members getting in, or is that not possible either?

Thanks

[DementedMindz]

hmm thats strange cause as soon as i submitted my site i had spiders crawl it and they showed up right away

[DementedMindz]

another question is there a way we can add more spiders? so they dont show up as unknown?

[samu2]

I think my site just hates me lately lol

The spiders have been stuck on the home page all day.I wonder if that has anything to do with it? mind you ad sense has been active.

[DementedMindz]

lol which spiders are they?

[samu2]

Google and slurp have been on and seems to have gotten stuck lol

I uploaded buttons and saw them three days after everyone else could.Had to put up a redirect last night,works ok for everyone else,bar me.

[DementedMindz]

thats really strange im not sure at all i would ask someone who knows alot more then me i never see mine on site long at all

[Zia]

Hello....we are getting spider...most of them are unown..
confused


http://www.golpo.net/forum/spiders.php



*Unknown Spider
Dillo/0.8.5 01-09-2006
10:49 PM 221.135.208.159

*Unknown Spider
Googlebot-Image/1.0 01-07-2006
01:09 AM 66.249.65.202

*Unknown Spider
Microsoft Data Access Internet Publishing Provider Protocol Discovery 01-07-2006
01:48 PM 203.189.231.13

*Unknown Spider
Mozilla/2.0 (compatible; MS FrontPage 5.0) 01-08-2006
02:35 PM 203.189.231.13

*Unknown Spider
Mozilla/3.01 (compatible 1-08-2006
10:50 PM 12.20.58.68, 69.88.30.146, 202.52.197.55, 202.52.197.56, 210.50.143.21

*Unknown Spider
Mozilla/4.0 (compatible; Google Desktop) 01-08-2006
05:11 PM 70.236.181.179

*Unknown Spider
Mozilla/4.0 (compatible; MSIE 6.0) 01-09-2006
10:58 PM 202.63.231.150

*Unknown Spider
MSFrontPage/5.0 01-07-2006
01:48 PM 203.189.231.13

*Unknown Spider
Nokia6600/1.0 (5.27.0) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0 01-08-2006
11:25 AM 202.56.4.109


Seemd enugh of unknown spiders..confused any of them bandwidth theaf or not?

Any one can say which bots they are ?


Thanx

[XtremeOffroad]

Installed fine on 3.5.3 but no spiders are showing.
Been watching them on who's online though.
Any Idea why this isnt seeing them?

[mikelbeck]

Thanks for that, is there anyway you can password protect the page for now at all to prevent members getting in, or is that not possible either?

Thanks
You can do that with an .htaccess file. GIS for "password protect page htaccess".

For everybody with the "unknown" spiders. If the spider's user agent isn't in the vbulletin_spider.xml file, then it's going to be reported as "unknown". You can take a whack at adding the spiders to that file yourself, or just keep an eye on the thread related to that file and wait for it to be updated.

[Devil Woman]

Lol I know many things but I wouldn't know where to start with that, are you familiar with that sort of thing at all and if I pm'd would you be able to help at all?

would be greatly appreciated

Thanks

[Lizard King]

I had to remove this because this script cannot track my spiders activity :) The log is too big and i always receive Maximum execution time of 30 seconds exceeded error with this.

[DementedMindz]

You would have to add more names of the bots in the spiders_vbulletin.xml file.
PHP Code:
<spider ident="METATAG INFO">
<name>SpiderName of Your Desire</name>
</spider>

[mikelbeck]

Lizard King: On what page do you get that message? And what do you mean, "the log is too big"? There is no log, it's a table in the database.

Devil Woman: Have a look at this page: http://www.4webhelp.net/tutorials/misc/htaccess.php

[Lizard King]

Lizard King: On what page do you get that message? And what do you mean, "the log is too big"? There is no log, it's a table in the database.

Devil Woman: Have a look at this page: http://www.4webhelp.net/tutorials/misc/htaccess.php

When i tried to view spiders.php the first day it worked but after first day it never worked. ıt might be because i have a lot of spider activity lately. Checked the attached log because of this i always get timeout errors.

[NexDog]

This is great - installs. I want admins only to be able to see this so what conditional would I need for forumhome...if condition="$bbuserinfo['user.....?

[armagan]

When i tried to view spiders.php the first day it worked but after first day it never worked. ıt might be because i have a lot of spider activity lately. Checked the attached log because of this i always get timeout errors.

!!!!

[mikelbeck]

This is great - installs. I want admins only to be able to see this so what conditional would I need for forumhome...if condition="$bbuserinfo['user.....?

Yeah, something like this:


<if condition="$permissions['adminpermissions']">
<a href="$vboptions[homeurl]/spiders.php">Spider Activity</a>
</if>

[Zia]

Google and slurp have been on and seems to have gotten stuck lol

I uploaded buttons and saw them three days after everyone else could.Had to put up a redirect last night,works ok for everyone else,bar me.

ahha...same here...Tahoo !Slurp just sticky my place...even 3/4 at a time..
Google very frequently...

just wondering too many spider can slow down forum ??

[NexDog]

Yeah, something like this:


<if condition="$permissions['adminpermissions']">
<a href="$vboptions[homeurl]/spiders.php">Spider Activity</a>
</if>

I did it with a usergroup ID:


<if condition="is_member_of($bbuserinfo, 6)">


6 being the ID of the admin usergroup. Can you think of any drawbacks to doing it that way?

The hack is so cool though. Since I installed just a few hours ago it's logged over 900 hits from google and 400 from slurp. :)

[Club3G]

Installed, but getting the following error in 3.5.3:


Database error in vBulletin 3.5.3:

Invalid SQL:
START TRANSACTION;

MySQL Error : You have an error in your SQL syntax near 'START TRANSACTION' at line 1
Error Number : 1064
Date : Thursday, January 12th 2006 @ 01:18:47 PM
Script : http://www.sevenstring.org/forum/spiders.php
Referrer :
IP Address : xxx
Username : Chris
Classname : vb_database


:(

[mikelbeck]

NexDog: I'm able to view your spiders page as a guest, so I'm going to assume that your code isn't working properly.

Club3G: What database & version are you using?

[Club3G]

Mysql version is 3.23.58, PHP version is 4.3.11 running on Linux.

(Not sure what you mean by which DB, though).

Edit: Thanks for the quick reply though. :D Looks like a killer hack, and I can't wait to get it working.

Also, I'm not sure if it matters, but there was no spiders.php at all in my forums root.

[mikelbeck]

Mysql version is 3.23.58, PHP version is 4.3.11 running on Linux.

(Not sure what you mean by which DB, though).

I meant "MySQL", you answered the question. ;-)

I'll have to check if MySQL 3 supports transactions, I have a feeling by the error you've gotten that it doesn't.


Also, I'm not sure if it matters, but there was no spiders.php at all in my forums root.

You have to put it there manually (ftp it to the directory).

[Club3G]

Aah, awrighty. :)

* Club3G anxiously overs over the Install button!

[mikelbeck]

Aah, awrighty. :)

* Club3G anxiously overs over the Install button!

You may want to find something else to do for a while, I'm not going to be able to have a look at it until later tonight or tomorrow. ;-)

[NexDog]

NexDog: I'm able to view your spiders page as a guest, so I'm going to assume that your code isn't working properly.

Club3G: What database & version are you using?
But no-one else apart from admins can see the link on the left column. :)

[Club3G]

/me is still poised over the install button, anxiously hoping that his DB error goes away

[mikelbeck]

* Club3G is still poised over the install button, anxiously hoping that his DB error goes away

Soon... Some things have come up in my life recently that require my full attention, this is going to have to be on hold for a little while. But I promise I will get back to it soon and will have something that will work for you.

[docvader]

Mikelbeck, I just installed this, and it's great. Within minutes, the Yahoo slurp spidered my site. Yahoo! Slurp 20 Hours Ago
-20:18
It's now 2023 hrs. And I installed it ten minutes ago.
Something wrong with the "hours" code, as was mentioned before. Hope this helps. Good luck with the "life issues" man, I know how that goes.

Thanks!
rich

[Zia]

just curious who got howmay hit & which spider?

[Club3G]

Soon... Some things have come up in my life recently that require my full attention, this is going to have to be on hold for a little while. But I promise I will get back to it soon and will have something that will work for you.

No problem at all bro, I appreciate the effort. Take care of yourself.

[mikelbeck]

Mikelbeck, I just installed this, and it's great. Within minutes, the Yahoo slurp spidered my site.
It's now 2023 hrs. And I installed it ten minutes ago.
Something wrong with the "hours" code, as was mentioned before. Hope this helps. Good luck with the "life issues" man, I know how that goes.

Thanks!
rich

Very odd.

Try this, see if it makes a difference...

Open up spiders.php, find this block of code:


$formatted_time = vbdate($vbulletin->options['timeformat'], strtotime(date("g:i A", mktime ($hour, $minute, $second, $month, $day, $year))), true);
$formatted_date = vbdate($vbulletin->options['dateformat'], strtotime(date("M j, Y", mktime ($hour, $minute, $second, $month, $day, $year))), true);
return ($formatted_date . '<br />' . $formatted_time);


Right before it, add this:


return ($datestring);


Then view your spiders page again and see if the dates are correct. If they are, then there's an issue with that block of code that I'll have to work out. If they're NOT, then it's something to do with MySQL.

In either case, post your results here.

[docvader]

Well, that made a big difference. Now it returns a date and time of last visit, such as "2006-01-16 03:59:17 6", instead of "x hours ago"

Just for your information, today, one day after I installed this (?), it went from "20 hours ago" to "19 hours ago". As if it's confusing the days for hours.

I'll keep an eye on this and see what happens tomw.

[mikelbeck]

Well, that made a big difference. Now it returns a date and time of last visit, such as "2006-01-16 03:59:17 6", instead of "x hours ago"

Just for your information, today, one day after I installed this (?), it went from "20 hours ago" to "19 hours ago". As if it's confusing the days for hours.

I'll keep an eye on this and see what happens tomw.

But is the "2006-01-16 03:59:17" that it's now displaying the correct time & date for your current time zone?

[docvader]

I am assuming so. The last spider that came through was at 2006-01-16 18:58:53, which was an 21 minutes ago. I do not know the exact time it actually visited me however.

Looks like you fixed it.

[darksoulz]

I would just like to chime in with the 'me too' on the problem of it showing Yesterday when it should say Today. I dont know if it might be interfering, but we also have this hack (https://vborg.vbsupport.ru/showthread.php?t=96505) installed.

On another note, is there any easy way to get it to not log the unknown spiders? It seems to be picking up regular users that are using non-standard user-agent strings and filling up the database to the point of making us hit php's execution time limit when trying to view the page.

[Zia]

Hello..
here is a idea..
spider watcher now listing spider in two section
1.Search Spider
2.Unknown spider.

Els that there are Forum Spider....
like boardreader www.boardreader.com
Its been listed in unknown spider tho its listed in list.xml

any way that list will show in 3 section Seach spider,Forum Spider , Unknwon Spider..

while spider have its address in the list..so if it possble the spider's name is Klik Able ..

Thanks

[mikelbeck]

I am assuming so. The last spider that came through was at 2006-01-16 18:58:53, which was an 21 minutes ago. I do not know the exact time it actually visited me however.

Looks like you fixed it.

No, not yet, but at least I know what's causing it...

[mikelbeck]

For those who are having the problem with the dates & times... What is your setting for "Format for Time" and "Format for Date" in the "Date and Time Options" in your admin control panel? Also, what setting do you have for "Datestamp Display Option"?

I believe if you change these lines of code in spiders.php:


$formatted_time = vbdate($vbulletin->options['timeformat'], strtotime(date("g:i A", mktime ($hour, $minute, $second, $month, $day, $year))), true);
$formatted_date = vbdate($vbulletin->options['dateformat'], strtotime(date("M j, Y", mktime ($hour, $minute, $second, $month, $day, $year))), true);


to be:


$formatted_time = vbdate($vbulletin->options['timeformat'], strtotime(date("g:i A", mktime ($hour, $minute, $second, $month, $day, $year))));
$formatted_date = vbdate($vbulletin->options['dateformat'], strtotime(date("M j, Y", mktime ($hour, $minute, $second, $month, $day, $year))));


...this problem should go away. If somebody could give that a shot and let me know I'd appreciate it.

[docvader]

Just curious, what's the problem? Mine seems to be working correctly with that little fix from yesterday. http://www.russbo.org/vb/spiders.php?

rich

[Brandon Sheley]

works on 3.5.2 patched to 3.5.3
* Brandon Sheley clicks install

http://locoforum.com/forums/spiders.php

[MPDev]

I used it for a little while, but the list on the spiders page of "unknown" spiders was getting HUGE; even after installing the recommended spider list xml file. I must have had 100+ unknown spiders listed.

[darksoulz]

For those who are having the problem with the dates & times... What is your setting for "Format for Time" and "Format for Date" in the "Date and Time Options" in your admin control panel? Also, what setting do you have for "Datestamp Display Option"?

On ours,
Format for date - m-d-Y
Format for time - h:i A
Datestamp display option is Yesterday/Today, but like I said in my previous post, we have a hack installed that makes it user configurable.

I'm at work right now and dont have access to the machine to edit the php file. I will do that when I get home tonight and report back if someone doesn't beat me to it.


Edit: Just noticed something, The problem seems to only be while I'm logged in. If i log out and view the page as an unregistered user, it displays correctly. Another thing i just noticed while digging deeper, its not just the yesterday/today thing that's not correct. Everything is a day off while logged in. One of the spiders shows as 01-14-2006 03:24 PM on the page, but in the database the timestamp is 2006-01-15 16:24:51. My timezone setting on the board is an hour off the default which would explain the time difference, but I cant figure out why its one off on the day everywhere.

http://gprime.net/board/spiders.php

[DementedMindz]

I used it for a little while, but the list on the spiders page of "unknown" spiders was getting HUGE; even after installing the recommended spider list xml file. I must have had 100+ unknown spiders listed.


well how come you dont add the spiders to your spiders.xml file? then you wouldnt have that....

[darksoulz]

If they were seeing the same thing we are, quite a few of the 'Unknown Spider' entries aren't spiders at all. They are humans with browsers that have a useragent that isn't being recognized.

[mikelbeck]

Edit: Just noticed something, The problem seems to only be while I'm logged in. If i log out and view the page as an unregistered user, it displays correctly. Another thing i just noticed while digging deeper, its not just the yesterday/today thing that's not correct. Everything is a day off while logged in. One of the spiders shows as 01-14-2006 03:24 PM on the page, but in the database the timestamp is 2006-01-15 16:24:51. My timezone setting on the board is an hour off the default which would explain the time difference, but I cant figure out why its one off on the day everywhere.

http://gprime.net/board/spiders.php

Do you still have the same issue if you apply the mod in this post:

https://vborg.vbsupport.ru/showpost.php?p=875905&postcount=98

?

[DementedMindz]

the only thing that i see so far is just the high sql queries... i think this wouldnt be to good on a large site in which spiders visit alot... is there any way to cut down on them? cause to be honest i only have my site in testing and its already using 70 queries with only 12 spiders that visited so far

[DementedMindz]

just for a example on a board with alot of spiders visiting i was just looking at darksoulz post about there site http://gprime.net/board/spiders.php look at how many sql queries it has and its alot .... Page generated in 6.39825 seconds with 1909 queries

[Brandon Sheley]

ouch, mine is up to 101 qeries.. I'll have to shut this down unless this is resolved :(

[Totti]

gprime is even more now:
Page generated in 16.68188 seconds with 4357 queries#
LOL!!!

i got 188 querries ... but i thought about leaving the link private and when the site is called just 12 times a week i can our server can handle this ... wouldn't be really great, though when this will be fixed!

[mikelbeck]

just for a example on a board with alot of spiders visiting i was just looking at darksoulz post about there site http://gprime.net/board/spiders.php look at how many sql queries it has and its alot .... Page generated in 6.39825 seconds with 1909 queries
Yeow... I just ran that page, it was 12,000 queries.

How do you enable the page generation time for each page? Or is that a hack?

That's way too much, lemme go back through the code and see what I can do with it!

[Totti]

How do you enable the page generation time for each page? Or is that a hack?
search for microstats on this board

[StarBuG]

Fatal error: Maximum execution time of 30 seconds exceeded in /path/forum/spiders.php on line 70

Guess it is not usable for me either but it is a nice idea.

Needs some improvements ;)

[DementedMindz]

How do you enable the page generation time for each page? Or is that a hack?


here is the link for the plugin https://vborg.vbsupport.ru/showthread.php?t=82900

[Detomah]

I'm seriously thankful, yet at the same time wishing to god I hadn't installed this hack a few days ago.....

I've been monitoring some of the spiders that visit and i've got a bit of a shock through the addon.

Some of the visitors I thought were guests turn out to be rogue spiders, such as email address hunters, site grabbers and the likes.

I hadn't spotted them in the past, so never knew they were visiting, I just thought they were guests, as that's what it comes up as.

Anyway, after a bit of tweaking and after finding this site:

http://www.kloth.net/internet/badbots.php

Which displays a pretty good list of bad bots and even shows you a couple of good ways to stop them.

I'm now getting a much smoother image of who my visitors are and how many of them are actually real people.

So...

Thanks a bunch for this hack, I wouldn't have believed it would come in so handy. :)

The page is getting pretty big, pretty fast on my site though and I'm sure it's gonna start timing out soon.

Is there any way you could make it, so it runs over multiple pages instead or something like that, to try and cut down the queries?

[DementedMindz]

Detomah reading what you just posted and from reading what he says are good and bad bots is funny.... they say them bots are bad? well strange look at his robots.txt lol he only has one bot in there lol so they must not be too bad huh? http://www.kloth.net/robots.txt

[darksoulz]

Do you still have the same issue if you apply the mod in this post:

https://vborg.vbsupport.ru/showpost.php?p=875905&postcount=98

?
hopefully this doesn't turn out too ugly...


mysql> select bot,timestamp from bots where bot like '%Find%';
+-----------------------------------------------------------------+---------------------+
| bot | timestamp |
+-----------------------------------------------------------------+---------------------+
| findlinks/1.1-a8 (+http://wortschatz.uni-leipzig.de/findlinks/) | 2006-01-20 13:06:27 |
+-----------------------------------------------------------------+---------------------+
1 row in set (0.02 sec)


but on the spiders.php page:
FindLinks 01-19-2006 12:06 PM

[Detomah]

Detomah reading what you just posted and from reading what he says are good and bad bots is funny.... they say them bots are bad? well strange look at his robots.txt lol he only has one bot in there lol so they must not be too bad huh? http://www.kloth.net/robots.txt

Yeah I had a look at that before and it amused me a little.

I think he may be simply failing to practice what he preaches.

Although he did say himself, some of the "bad bots" he mentioned aint actually that bad.

[nitro]

Yeah I had a look at that before and it amused me a little.

I think he may be simply failing to practice what he preaches.

Although he did say himself, some of the "bad bots" he mentioned aint actually that bad.

Nope you both failed to read. It is pointless using robots.txt to dissallow bots that simply ignore or missuse the robots.txt standard. The majority of the bots in that list have the code "M" or "N" which means to ignore or missuse robots.txt

Hence he has a bot trap to deal with them instead. ;)

[Brandon Sheley]

any word on dropping the queries down ?

[mikelbeck]

any word on dropping the queries down ?
Not yet, I haven't had a chance to get back to this. I will try to have a look at it this evening or tomorrow morning.

[Brandon Sheley]

good luck :)

[Detomah]

Nope you both failed to read.

Erm, sorry, but I didn't fail to read anything, he actually mentions the bot trap on the same page and even provides a link to said bot trap:

http://www.kloth.net/internet/bottrap.php

That was the main reason I posted the link to the page and it was the first thing I actually created for myself after reading the article.

Hence he has a bot trap to deal with them instead. ;)

I know... :p

I thought it made sense to provide a link to something that would give a little better understanding of some of the spiders, for those installing this script that's all, I was only trying to be helpful, I just got myself a little confused with that reply above, after all i'm normally posting on here at 4-5 am after being up 18-20hrs. :(

[nitro]

Erm, sorry, but I didn't fail to read anything, he actually mentions the bot trap on the same page and even provides a link to said bot trap:

http://www.kloth.net/internet/bottrap.php

That was the main reason I posted the link to the page and it was the first thing I actually created for myself after reading the article.


Sorry didnt mean it to sound in a bad way at all. ;)


I know... :p

I thought it made sense to provide a link to something that would give a little better understanding of some of the spiders, for those installing this script that's all, I was only trying to be helpful, I just got myself a little confused with that reply above, after all i'm normally posting on here at 4-5 am after being up 18-20hrs. :(

I would guess most who post here do that more than once :D

The link was very useful and thanks for posting it. :)
I can verify the the bot-trap dir and blacklist.dat file works quite nicely. I have yet to implement the full function of the blacklist.dat for blocking to vb tho. Im not sure of the best way to do this. perhaps a simple include hook in global_start would work ?

[Megareus Rex]

Great hack, but I'm having the problem where it doesn't display my board's default timezone offset.

Is there a way to set it so it uses a user's offset preference?

[mikelbeck]

I working on version 1.0.0 Beta 4 right now. It should fix the time/date issue, a possible SQL injection exploit, and the high number of queries. I'm letting it run on one of my sites now to gather up a bunch of spider data so I can see if the number of queries drops. If it does, I'll package it up and will post an update tonight or tomorrow morning.

In the mean time, somebody notified me via e-mail that the plug-in can be exploited by a malicious user... So, for the time being, if you'd like to diable the "Spider Watcher" plug-in (under the global_start hook location) you can do that and you'll be safe. Or you can either wait for the update or modify the code yourself.

Here's the updated code for the plug-in:


if (!$vbulletin->userinfo['userid'])
{
$ip_address_parts = explode(".", $_SERVER['REMOTE_ADDR']);
$ip_address = $ip_address_parts[0] * pow(256,3) + $ip_address_parts[1] * pow(256,2) + $ip_address_parts[2] * pow(256,1) + $ip_address_parts[3] * pow(256,0);

$write_bot = $db->query_first("INSERT INTO " . TABLE_PREFIX . "bots (bot, ip_address, page, timestamp) VALUES ('" . $vbulletin->db->escape_string(trim($_SERVER['HTTP_USER_AGENT'])) . "','" . $ip_address . "','" . THIS_SCRIPT . "', NOW())");
}

[the Sandman]

I looked at your online demo and I didn't see any mention of activity in the archive. Does this script not track spiders there?

[mikelbeck]

Version 1.0.0 Beta 4 is now available.

- Corrected potentional SQL injection issue in plug-in
- Reduced the number of SQL queries required to display statistics
- Corrected date/time display issue

[silurius]

Upgraded w/o issues.

[mikelbeck]

Upgraded w/o issues.

Are the times & dates being displayed properly for you?
Have the number of queries decreased for you?

[Guest210212002]

* Guest210212002 still can't install this.

I'm this guy. (https://vborg.vbsupport.ru/showpost.php?p=871700&postcount=79)

[XtremeOffroad]

Getting this error
Warning: array_multisort(): Argument #1 is expected to be an array or a sort flag in /spiders.php on line 223

And still no spiders show on my site.
I had asked for help with this in the past with no reply.

[Zia]

sounds nice.....
this detect search sipder & unknown spider.....but how to show the bot's home page url in the bots list page(generated page) bellow the botnick. ?

Just curious any one tried to detect unknown spiders??

thnx

[Megareus Rex]

A concern and suggestion.

One of the spiders (Yahoo! Slurp) is returning literally DOZENS of IPs. Here's the particular block I have now (and I've only had it for a couple days):

0.0.0.115, 68.142.249.15, 68.142.249.19, 68.142.249.20, 68.142.249.21, 68.142.249.25, 68.142.249.31, 68.142.249.35, 68.142.249.38, 68.142.249.42, 68.142.249.44, 68.142.249.47, 68.142.249.48, 68.142.249.58, 68.142.249.61, 68.142.249.67, 68.142.249.68, 68.142.249.73, 68.142.249.81, 68.142.249.83, 68.142.249.84, 68.142.249.85, 68.142.249.91, 68.142.249.92, 68.142.249.96, 68.142.249.98, 68.142.249.99, 68.142.249.102, 68.142.249.108, 68.142.249.110, 68.142.249.112, 68.142.249.115, 68.142.249.116, 68.142.249.117, 68.142.249.119, 68.142.249.120, 68.142.249.124, 68.142.249.127, 68.142.249.132, 68.142.249.152, 68.142.249.154, 68.142.249.159, 68.142.249.164, 68.142.249.168, 68.142.249.176, 68.142.249.188, 68.142.249.191, 68.142.249.201, 68.142.249.207, 68.142.249.208, 68.142.250.11, 68.142.250.12, 68.142.250.13, 68.142.250.14, 68.142.250.15, 68.142.250.22, 68.142.250.26, 68.142.250.28, 68.142.250.36, 68.142.250.43, 68.142.250.53, 68.142.250.65, 68.142.250.73, 68.142.250.77, 68.142.250.79, 68.142.250.83, 68.142.250.86, 68.142.250.91, 68.142.250.93, 68.142.250.101, 68.142.250.102, 68.142.250.111, 68.142.250.114, 68.142.250.116, 68.142.250.118, 68.142.250.119, 68.142.250.122, 68.142.250.124, 68.142.250.126, 68.142.250.130, 68.142.250.131, 68.142.250.141, 68.142.250.142, 68.142.250.147, 68.142.250.152, 68.142.250.153, 68.142.250.154, 68.142.250.155, 68.142.250.158, 68.142.250.163, 68.142.250.165, 68.142.250.167, 68.142.250.169, 68.142.250.172, 68.142.250.176, 68.142.250.180, 68.142.250.181, 68.142.250.183, 68.142.250.187, 68.142.250.189, 68.142.250.193, 68.142.250.199, 68.142.250.202, 68.142.250.203, 68.142.250.208, 68.142.251.14, 68.142.251.18, 68.142.251.19, 68.142.251.23, 68.142.251.25, 68.142.251.34, 68.142.251.46, 68.142.251.47, 68.142.251.59, 68.142.251.69, 68.142.251.81, 68.142.251.85, 68.142.251.86, 68.142.251.92, 68.142.251.96, 68.142.251.101, 68.142.251.110, 68.142.251.113, 68.142.251.119, 68.142.251.123, 68.142.251.129, 68.142.251.132, 68.142.251.144, 68.142.251.153, 68.142.251.154, 68.142.251.155, 68.142.251.159, 68.142.251.167, 68.142.251.170, 68.142.251.180, 68.142.251.184, 68.142.251.185, 68.142.251.190, 68.142.251.196, 68.142.251.201, 68.142.251.203, 202.160.180.127

You might want to consider adding an IP display limit with a "View All" link to see all the IPs at your request, rather than displaying all on the main page itself.

[mikelbeck]

Chris-777: Still working on that. Don't worry, I haven't forgotten about you.

XtremeOffroad: That sounds like there's nothing to be sorted, so the multisort is failing. I will put a check in there in the next version. As for no spiders... Do they visit your site? Are you sure that the plug-in is installed and active?

Zia: What if I made the spider's name a hotlink instead of displaying the URL under it?

Megareus Rex: I'll see what I can do with that. Maybe just display 5 or so with a "view others" link to show the rest? Keep in mind that only users who have the "view IP" privliledge (such as admins or moderators) will see the IPs. All other users see nothing.

Everybody: What about the number of queries?

[Guest210212002]

:D Thanks dude. If it's just me that has the problem, no worries, I'll get by. It's a fantastic hack otherwise.

/salute

[Megareus Rex]

Chris-777: Still working on that. Don't worry, I haven't forgotten about you.

XtremeOffroad: That sounds like there's nothing to be sorted, so the multisort is failing. I will put a check in there in the next version. As for no spiders... Do they visit your site? Are you sure that the plug-in is installed and active?

Zia: What if I made the spider's name a hotlink instead of displaying the URL under it?

Megareus Rex: I'll see what I can do with that. Maybe just display 5 or so with a "view others" link to show the rest? Keep in mind that only users who have the "view IP" privliledge (such as admins or moderators) will see the IPs. All other users see nothing.

Everybody: What about the number of queries?

A "View All" link would definitely be good. I know only those with IP viewing perms can see them, but I'm one of those people, and its rather annoying to see the Yahoo! Slurp spider growing ever larger... :P

As for the # of queries....well....just check this link out (look at the bottom):
http://www.evermoreforums.com/forums/spiders.php

I've been getting 800+ queries, though it seems to be growing (was only 700+ a few hours ago). So yeah...loads.

[mikelbeck]

I've been getting 800+ queries, though it seems to be growing (was only 700+ a few hours ago). So yeah...loads.

Yeah, that's way too much. I'll have to take another look at that.

I know how to fix it, the problem is I don't want to do it that way. What happens is this... When a guest arrives at a page, the spider plug in writes a line to the database. Just the timestamp, user agent and page. When you run the spiders page, it "rolls up" all of the data so it's displayed in a nice, neat format and all the old data is removed. That's what's creating so many queries.

If I were to change the plug in to check to see if there's already a record for that spider in the database and then just update it as the page is loaded, there would be no need to "roll up". But I think that would add a few queries on each page load (by a non-user, meaning a guest or spider) and I don't think that's a good idea.

I'll see what else I can come up with.

[Megareus Rex]

Yeah, that's way too much. I'll have to take another look at that.

I know how to fix it, the problem is I don't want to do it that way. What happens is this... When a guest arrives at a page, the spider plug in writes a line to the database. Just the timestamp, user agent and page. When you run the spiders page, it "rolls up" all of the data so it's displayed in a nice, neat format and all the old data is removed. That's what's creating so many queries.

If I were to change the plug in to check to see if there's already a record for that spider in the database and then just update it as the page is loaded, there would be no need to "roll up". But I think that would add a few queries on each page load (by a non-user, meaning a guest or spider) and I don't think that's a good idea.

I'll see what else I can come up with.

Just letting you know, its almost at 1100 queries, and a page generation time of 7+ seconds.

Not to mention the Yahoo Slurp spider's # of IPs has more than doubled (perhaps close to tripled) from earlier.

[Brandon Sheley]

Are the times & dates being displayed properly for you?
Have the number of queries decreased for you?


yes, queries are much lower.. under 100

not sure how to check time ? is there a way to reset,, so it shows zero spiders viewed ?

this would be handy i think..

so i could reset once a week, or a month,, to see how the spider traffic is...

thank you,, good hack still..

* Brandon Sheley upgraded with no problems

http://locoforum.com/forums/spiders.php
Page generated in 0.22789 seconds with 79 queries

[Megareus Rex]

And now my queries are up to 1750....

[Totti]

ye, would be cool ... as i added google-feedfetcher to my spiderlist (wasn't in it :( )yesterday
and now i have an unkown entry with the old visits and a new known entry with the visits since yesterday :D

[Megareus Rex]

Now its down to like 1350 or so. (# of queries)

[Robbed]

Database error in vBulletin 3.5.3:

Invalid SQL:
START TRANSACTION;

MySQL Error : You have an error in your SQL syntax near 'START TRANSACTION' at line 1
Error Number : 1064
Date : Thursday, January 26th 2006 @ 06:10:48 PM
Script :
Referrer :
IP Address :
Username :
Classname :
Any fix for this?

[XtremeOffroad]

XtremeOffroad: That sounds like there's nothing to be sorted, so the multisort is failing. I will put a check in there in the next version. As for no spiders... Do they visit your site? Are you sure that the plug-in is installed and active?

Yea they do visit quite often, Yes the plugin is active, I've had the issue since the first release of this hack.
Kept it installed in hopes of a fix.

Thanks for the reply, Fingers crossed for a fix :nervous:

[mikelbeck]

Yea they do visit quite often, Yes the plugin is active, I've had the issue since the first release of this hack.
Kept it installed in hopes of a fix.

Thanks for the reply, Fingers crossed for a fix :nervous:
Can you take a look in your vb_bots table and see if there's any data in it?

[XtremeOffroad]

Can you take a look in your vb_bots table and see if there's any data in it?

Where is the vb_bots table?

[Brandon Sheley]

that would be in your MySQL

[XtremeOffroad]

that would be in your MySQL

I thaught so but I didnt even see it there.

Edit: I do not have vb_bots but do have a Bots which contains

Field Type Collation Attributes Null Default Extra Action
id int(11) No auto_increment
bot varchar(255) latin1_swedish_ci No
nice_bot varchar(255) latin1_swedish_ci No
timestamp datetime Yes 0000-00-00 00:00:00
ip_address varchar(255) latin1_swedish_ci No
page varchar(255) latin1_swedish_ci No
rollup int(11)


Space usage:
Type Usage
Data 0 Bytes
Index 1,024 Bytes
Total 1,024 Bytes

If this is what you were referring to.

[mikelbeck]

I thaught so but I didnt even see it there.

Edit: I do not have vb_bots but do have a Bots which contains

Field Type Collation Attributes Null Default Extra Action
id int(11) No auto_increment
bot varchar(255) latin1_swedish_ci No
nice_bot varchar(255) latin1_swedish_ci No
timestamp datetime Yes 0000-00-00 00:00:00
ip_address varchar(255) latin1_swedish_ci No
page varchar(255) latin1_swedish_ci No
rollup int(11)


Space usage:
Type Usage
Data 0 Bytes
Index 1,024 Bytes
Total 1,024 Bytes

If this is what you were referring to.

Yes.

Do the rest of your tables have any sort of prefix? (like vb_ or something similar)

There's no data in that table, that's why it's not showing you any data.

[XtremeOffroad]

Yes.

Do the rest of your tables have any sort of prefix? (like vb_ or something similar)

There's no data in that table, that's why it's not showing you any data.

No There is no (VB_) prefix on any of the tables.

How would I go about getting data to show?
And thank you very much for the replies to help.
Hopefully there is an easy fix.

[Detomah]

You might want to consider adding an IP display limit with a "View All" link to see all the IPs at your request, rather than displaying all on the main page itself.

I too would love to see something like this implemented, as Yahoo Slurp is awefully prolific and seems to be using a different IP address every single visit and at 17,000+ visits, it's getting pretty damn long at the moment.

[Megareus Rex]

Just another query update: Its now up to anywhere from 2200 to 4000 queries >.<

[StarBuG]

Any solution in sight for larger forums?

[mikelbeck]

Yes, I'm close to having a new version ready... One that should fix the heavy query issue.

[ChuanSE]

which is the hack you use on your forum to state winxp like the users didn't register yet mikelbeck? ya know, the yellow bar on top of the browserscreen in yellow

[Zia]

hello

Its My Spider's Page Listing Status

Page generated in 2.55627 seconds with 394 queries [Server Loads: 1.61 2.18 : 2.45]
[Page Compression: 42.06 k/45.78 k (8.13%)]

michelbeck..
can u add a modification in ur next release..

that now its devided in tow part

like the ip-address showing all..& it making the page too long..too long..just like Yahoo!Slurp...hundreds of IP...
it will show Ip Range..least this shorten page..

& the 2 part known & unknow...here can u devide few more..like
Search/Forum/Feed & Links/
its just my idea..

Thankx

[Brandon Sheley]

Yes, I'm close to having a new version ready... One that should fix the heavy query issue.


can I make a request, that shouldn't be hard.. at least I wouldn't think it should..

I'm doing and working on a lot of SEO changes for my site, and something like this could help me see whats working and whats not...



.could u add something in vboptions that could "reset" the count ?

. show a more detailed page of where they are looking ?

showthread= ?

have this an options to show too.. cuz I'm sure larger sites wouldn't want all this information logged.. maybe this would just involve more work then u want to do..


but the reset would be great for me, or maybe if you could post a SQL query that I can run that would do this ?

thank you :)

[redspider]

Hello, could someone please tell me why i am getting this error?

Warning: array_multisort(): Argument #1 is expected to be an array or a sort flag in /spiders.php on line 223

Also, i have no Spiders showing and there are several on my site as we speak...Thank you for any help in advance -

[redspider]

Ok, i dont know what happened, but my Sider Watch just mysteriously started working...No more error, and the Spiders are showing just fine...Maybe it just takes some time to kick in lol who knows, who cares? it works now...AWESOME HACK! Thank you!

[mikelbeck]

Ok, here's where I'm at right now.

I've got version 1.0.0 Beta 5 running on one of my sites. I just ran the spider page for the first time today, it took 0.21342 seconds with 57 queries. That's quite a bit better. ;-)

The problem is that the plug-in is a little heavy, I'd rather not do it this way. It checks to make sure that the wol_spiders is available (that's what contains the spider user agents) and then updates the database with the correct information at that point, instead of when the spiders page is run. I'm not sure if the code that loads up wol_spiders runs on EVERY page load or not, if it does that's bad. If it doesn't it should be OK. I'm still working on it...

The way the data is written to the database has changed in this new version, so the data that's currently in the database will be wiped out when this version is installed. Sorry, there's no way around that.

I'm still working on it, I should be able to put this version out in another couple of days.

[XtremeOffroad]

Hello, could someone please tell me why i am getting this error?

Warning: array_multisort(): Argument #1 is expected to be an array or a sort flag in /spiders.php on line 223

Also, i have no Spiders showing and there are several on my site as we speak...Thank you for any help in advance -


I'm still having the same error.
Any ideas on a fix?
Thanks

[yellowchaser]

Hey, how did you do the bar at the top that comes up and says "Hi it appears you have not registered to our community, to register please click here" ??

[mikelbeck]

I'm still having the same error.
Any ideas on a fix?
Thanks
Can you PM me the info for your site? I'd like to have a look.

[mikelbeck]

Hey, how did you do the bar at the top that comes up and says "Hi it appears you have not registered to our community, to register please click here" ??

https://vborg.vbsupport.ru/showthread.php?t=97150

[XtremeOffroad]

Can you PM me the info for your site? I'd like to have a look.
mikelbeck you are absolutely amazing, I located the culpret that caused the problem and the hack is working beautifully now.
I cannot thank you enough for your involvement and help.
Some of the best support I've ever received.
:banana: :banana: :banana: :banana:

[mikelbeck]

mikelbeck you are absolutely amazing, I located the culpret that caused the problem and the hack is working beautifully now.
I cannot thank you enough for your involvement and help.
Some of the best support I've ever received.

No problem. Now send me all of your money!

:cheeky:

Glad to hear it's working for you... I'm sure some of the other hacks you had installed weren't working properly, either, since it was failing where it was.

[XtremeOffroad]

No problem. Now send me all of your money!

:cheeky:

Glad to hear it's working for you... I'm sure some of the other hacks you had installed weren't working properly, either, since it was failing where it was.

Strangely though they were working, All but one anyway.
Now I need to find out what one was failing er rather the correct name of the hach so I can find a fix.
Thanks again.

[Brandon Sheley]

mikelbeck,

Sorry to ask this again, since it seems I'm the only one requested this but...

is there a way I can reset these stats ?

as I stated in my last post, I'm trying/testing out several SEO techniques and I'd like to be able to reset these one week,, see how many spiders and where they are going,,
then make some SEO changes and reset again, and compare the results :)

if it's as easy and clearing out a MySQL table, please inform me what one, as I'm not a pro with SQL and wouldn't want to ruin something else my clearing out the wrong table..

and if you can add a reset option in vboption that would be prefect for me at least :)

thank you

[prettyboi]

What do they mean by "install it to the root of my forum"? What root forum? Where can I find this directory?

[nitro]

What do they mean by "install it to the root of my forum"? What root forum? Where can I find this directory?

the same place as where your forums main index.php resides.

root of ?? == the top most directory level of that item. :)

[mikelbeck]

mikelbeck,

Sorry to ask this again, since it seems I'm the only one requested this but...

is there a way I can reset these stats ?

as I stated in my last post, I'm trying/testing out several SEO techniques and I'd like to be able to reset these one week,, see how many spiders and where they are going,,
then make some SEO changes and reset again, and compare the results :)

if it's as easy and clearing out a MySQL table, please inform me what one, as I'm not a pro with SQL and wouldn't want to ruin something else my clearing out the wrong table..

and if you can add a reset option in vboption that would be prefect for me at least :)

thank you

You can just truncate the table. If you have phpMyAdmin, go to the bots table, there should be on option (usually on the right hand side) that says "empty". That'll get rid of all the data in the table but keep the structure.

[Brandon Sheley]

thank you,, that's what i figured, but wanted to make sure b4 I emptied something that I shouldn't

[Zia]

helo michelbeck..
in ur current listed page there are two block
Search & unknown

as i would like to req. u to add One More portion Forum Spider
www.boardreader.com
www.boardtracker.com [this bot is very harmfull..that can even access every are..]

we needed to know that our site is being listed there or not....

can u have a thought on it?

[mikelbeck]

I'll see if that's something I can do. I can't promise it, though...

[mikelbeck]

Version 1.0.0 Beta 5 is now available.

1.0.0 Beta 5 - 02/01/2006
- Reduced the number of SQL queries required to display statistics

***UPGRADE NOTE***
When you upgrade from version 1.0.0 Beta 4 to 1.0.0 Beta 5 your existing spider data will be lost!

[shaynehammy]

If my spiders page doesnt show any spiders does that mean my forum sucks ass and google cant be bothered with it?

[nitro]

If my spiders page doesnt show any spiders does that mean my forum sucks ass and google cant be bothered with it?

Perhaps its juts google is too busy sucking ass elsewhere :D

@mikelbeck I have just noticed having looked at the old vb 3 mod talkerbot to see if theres any easy way to get it to work in 3.5 that your spider watcher table 'bots' clashes with one of the main alice engine tables for it. Maybe it would be wise to change yours while in an early stage of develoment to a more relevant name ie: 'spiders' or 'spider_watcher' ?

I suspect it would be far easier for yours to change than the other way as the alice engine itself is a 3rd party engine to the mod itself.

[mikelbeck]

@mikelbeck I have just noticed having looked at the old vb 3 mod talkerbot to see if theres any easy way to get it to work in 3.5 that your spider watcher table 'bots' clashes with one of the main alice engine tables for it. Maybe it would be wise to change yours while in an early stage of develoment to a more relevant name ie: 'spiders' or 'spider_watcher' ?

I suspect it would be far easier for yours to change than the other way as the alice engine itself is a 3rd party engine to the mod itself.

Thanks for that info, I'll take care of that in the next release.

[nitro]

Thanks for that info, I'll take care of that in the next release.

Awsome cuz I have about sorted the old talkerbot to work I just havent figured how to add the extra option for random chance of a response to the forumoptions page in admincp and its not parsing its own smilies. :)

And Iv no doubt once ExtremeTim gets the full new version out as its supposed to be a multi bot the bots table will get used a lot more.

[BoardTracker]

www.boardtracker.com [this bot is very harmfull..that can even access every are..]
we needed to know that our site is being listed there or not....
BoardTracker is not 'harmful', it is a search engine for forums. Please take a closer look before making such assumptions. To know if your site is indexed (it isn't) simply search for it using 'site:domain.com'. BoardTracker is robots.txt compliant anyway so you can fully control access.

[Brandon Sheley]

there have been many discussions about boardtracker at TAZ, I suggest those who have question go check it out..

[C Braithwaite]

my list doesnt appear to be seperating it's self.. it just has the one "unknown Spider" item, and keeps changing the name of the spider depending on the most recent spider.

http://www.mished.co.uk/forum/spiders.php

I've checked the database, and spiders from google, yahoo and msn visit, but dont pop up in my list. ?

[mikelbeck]

my list doesnt appear to be seperating it's self.. it just has the one "unknown Spider" item, and keeps changing the name of the spider depending on the most recent spider.

http://www.mished.co.uk/forum/spiders.php

I've checked the database, and spiders from google, yahoo and msn visit, but dont pop up in my list. ?

What it's showing now is msnbot/1.0, which should be identified as a "known" spider.

Did you update the spiders list? (downloaded the new xml file and install it)

[C Braithwaite]

What it's showing now is msnbot/1.0, which should be identified as a "known" spider.

Did you update the spiders list? (downloaded the new xml file and install it)


yeah, i downloaded it, and double checked it just a while ago..

[C Braithwaite]

yeah, i downloaded it, and double checked it just a while ago..

as you may see, the last spider (depeneding on when you look) is now "Unknown Spider
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) 02-08-2006
02:25 AM "

[mikelbeck]

That's very strange...

Did you do a fresh install of 1.0.0 Beta 5 or did you upgrade from a previous version?

[hotwheels]

great job mikelbeck.........

[mikelbeck]

Beta 7 will be ready tomorrow or Friday morning. This version will fix the problem that C Braithwaite uncovered, plus will add the type of spider being identified (search engine spider, link checker, etc)

[Zia]

Beta 7 will be ready tomorrow or Friday morning. This version will fix the problem that C Braithwaite uncovered, plus will add the type of spider being identified (search engine spider, link checker, etc)


Dear Mike
it would be real smart..

if u dont mind once u can see my spiders page...
http://www.golpo.net/forum/spiders.php

Hope we will have all the classification.......& if it show the IP range instead of showing all the IP like Yahoo/Msn thay just made there home there :S lot lot of ip..make the page miles long....

Warm Wishes To Yu......

[StarBuG]

Hi

One request.
The Yahoo Slurp spider uses hundrets of different IPs that are listed in the spider list.
It is way to much to be usefull.
Would it be possible to have the ips grouped or limit the display to lets say 20-50 ips and the rest is only visible via a popup if needed?

Would be nice

[C Braithwaite]

Beta 7 will be ready tomorrow or Friday morning. This version will fix the problem that C Braithwaite uncovered, plus will add the type of spider being identified (search engine spider, link checker, etc)


It was just a fresh install, not an upgrade - glad you figured it out tho - what was the problem?

[mikelbeck]

It was just a fresh install, not an upgrade - glad you figured it out tho - what was the problem?
It was only saving the last "unknown" spider that it came across, and dumping all the ones before it. ;-)

[Mudvayne]

mikelbeck seems u forget to send us update notification mail :(.. today is 10th feb.. U updated ur hack on 1st feb :(

[C Braithwaite]

mikelbeck seems u forget to send us update notification mail :(.. today is 10th feb.. U updated ur hack on 1st feb :(


you can sign up for notification mail? how?

[Mudvayne]

you can sign up for notification mail? how?
u don need to signup for notification :S.. If u clickes install then after an update if coder clk the update button or something like this.. a mail has been sent to u automatically.. u just hav to click the install button :)

[C Braithwaite]

u don need to signup for notification :S.. If u clickes install then after an update if coder clk the update button or something like this.. a mail has been sent to u automatically.. u just hav to click the install button :)

ahh rite cool ^^ already done ^^

[mikelbeck]

mikelbeck seems u forget to send us update notification mail :(.. today is 10th feb.. U updated ur hack on 1st feb :(

Shuvo -

I did send an update when I updated the release information. Maybe you forgot to receive it? ;-)

I dunno, maybe there was an issue with it being sent from here. Whenever I post an update I also post a reply on this thread, so hopefully you would get a notification from this topic.

[Mudvayne]

ahh its ok.. thnx.. may b vb.org's mail server prob or accidently i delete it before chk..

[Zia]

:O friday :O

where is the new update release beta7 ???

[mikelbeck]

New version available...

This is an "interim" release, it just fixes the two items listed below. Other functionality will be in Beta 8, coming in the next couple of days.

1.0.0 Beta 7 - 02/11/2006
- Corrected issue with "unknown" spiders not being displayed properly.
- Added tracking of the type of spider (searchspider, link checker, etc)

[C Braithwaite]

New version available...

This is an "interim" release, it just fixes the two items listed below. Other functionality will be in Beta 8, coming in the next couple of days.

1.0.0 Beta 7 - 02/11/2006
- Corrected issue with "unknown" spiders not being displayed properly.
- Added tracking of the type of spider (searchspider, link checker, etc)


dont know if it was supposed to do this or not, but im not getting ANY spiders now... is it supposed to show the old spiders?

[Mudvayne]

dont know if it was supposed to do this or not, but im not getting ANY spiders now... is it supposed to show the old spiders?
Look up for instruction..
***UPGRADE NOTE***
When you upgrade from version 1.0.0 Beta 5 to 1.0.0 Beta 7 your existing spider data will be lost!

[mikelbeck]

dont know if it was supposed to do this or not, but im not getting ANY spiders now... is it supposed to show the old spiders?

The table was emptied during the upgrade... Give it a little while, they'll show up.

[C Braithwaite]

ahh sorry - i didnt see the upgrade notice - my table didnt empty btw - got 14+ pages of spiders in mysql.. just so you know. ill empty it manually

[mikelbeck]

ahh sorry - i didnt see the upgrade notice - my table didnt empty btw - got 14+ pages of spiders in mysql.. just so you know. ill empty it manually

If it wasn't showing any spiders after you install this version, it did empty the table. The stuff that's in the table now is what's been added since you did the upgrade.

BTW - here's what I'm planning for the next version:

- Split up the display of spiders by "type". So "searchspiders", "misc_spiders", "http_monitors" etc will each be in their own table.
- Reduce the number of IP addresses displayed, and add a link to a pop-up to show all of the IPs for a spider.
- Combine spiders that have the same name but different user agents. For example, on my site, Google shows up as "Googlebot/2.1 (+http://www.google.com/bot.html)" and "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)". This started happening after I fixed the "unknown spiders" issue in this release, it'll be fixed in the next one.
- See if I can make it work without transactions, so it'll work with earlier versions of MySQL.

I think that's it... If you've got any suggestions post them and I'll see if I can get them done.

[StarBuG]

Make the Botname a link if a link is specified in the spider.xml
So you can directly check what spider is crawling your forum if the info is available ;)

[C Braithwaite]

Still showing up as unknown :S http://www.mished.co.uk/forum/spiders.php

[Megareus Rex]

Just out of curiosity, why does the spider archive get cleared out every time this hack is upgraded?

[joeychgo]

Link Placement suggestion:


Place it in the footer - with a conditional so that only admins can see it:


In the footer template look for:


<if condition="$show['admincplink']"><a href="$admincpdir/index.php$session[sessionurl_q]">$vbphrase[admin]</a> -</if>


and below it add:

<if condition="$show['admincplink']"><a href="LINK TO YOUR SPIDERS.PHP FILE">Spiders</a> -</if>



You can put this same link anywhere else you would like, and only admins will see it.

[mikelbeck]

Thanks for posting that, Joey. Which of your sites are you running this on?

Megareus Rex - the tables only need to be cleared if there's an update to the tables.

[nicedreams]

There is a html coding error in the template.

There is a <thead> but not used or needed towards the top of the template.

Jim

[kenanulas]

how ı can ı put it into footer

[Mudvayne]

how ı can ı put it into footer
in ur footer template search for..
<a href="#top" onclick="self.scrollTo(0, 0); return false;">$vbphrase[top]</a>

& add just above.. ( for all user )
<a href="$vboptions[homeurl]/forum/spiders.php">Spider Activity</a>
or for just admin..
<if condition="$permissions['adminpermissions']"><a href="$vboptions[homeurl]/forum/spiders.php">Spider Activity</a> </if> |

[Murty]

If anyone is interested. If you would only like admins to be able to view the 'Spiders' link in the navbar. Do the following.

1. Locate the 'Navbar' template.
2. Find:

<td class="vbmenu_control"><a href="spiders.php$session[sessionurl_q]">Spiders</a></td>

3. Replace with:

<if condition="$show['admincplink']"><td class="vbmenu_control"><a href="spiders.php$session[sessionurl_q]">Spiders</a></td></if>

Should work.

[Mudvayne]

3. Replace with:

<if condition="$show['admincplink']"><td class="vbmenu_control"><a href="spiders.php$session[sessionurl_q]">Spiders</a></td></if>

Should work.

Yes.. It 'll work.. But its not the proper way I guess..

[nicedreams]

No known spiders have shown up since i've upgraded to beta 7.

Jim

[mikelbeck]

Beta 8 is now available.

1.0.0 Beta 8 - 02/19/2006
- Change the display of IP addresses to be a pop-up so they're all not displayed on the main page.
- Combined the spiders that have the same name but different user agents.

Upgrade Instructions
1. Upload (and overwrite) spiders to the root of your forum.
2. Import the file product-spiderwatcher.xml using the Manage Products module. Make sure the "Allow Overwrite" option is set to "Yes".
3. Add a link to spiders.php on your navbar or footer.

***UPGRADE NOTE***
When you upgrade from version 1.0.0 Beta 7 to 1.0.0 Beta 8 your existing spider data will be lost!

To make sure that you can decode the maximum amount of spiders, you should grab the latest spiderlist.xml and replace the spiders_vbulletin.xml file in your forumhome/includes/xml/ directory with the one from this thread: http://www.vbulletin.com/forum/showthread.php?t=76662

[mikelbeck]

There is a html coding error in the template.

There is a <thead> but not used or needed towards the top of the template.

Jim
This has been corrected in Beta 8. Thanks for letting me know about it.

[Guest210212002]

DB error importing the update.


Database error in vBulletin 3.5.3:

Invalid SQL:
INSERT INTO vb3_spider_watcher (nice_bot, bot, ip_address, page, type, timestamp) VALUES ('Unknown Spider','Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1','1113786890','showthread','', NOW());

MySQL Error : Table mydatabasename-removed.vb3_spider_watcher' doesn't exist
Error Number : 1146
Date : Sunday, February 19th 2006 @ 02:54:15 PM
Username : Unregistered
Classname : vb_database

[mikelbeck]

DB error importing the update.
When you installed it, did you "allow overwrite"?

Can you try to re-install it?

[Guest210212002]

It's still working, I just got the two emails about it. I allowed overwrite as well. Reinstall gives me the same thing.

It's not a big issue, since it's still working, just figured it was worth mentioning. ;)

[Zia]

yeah micklebeck...

its sober & nice upgrade..least it makes the makes page miles long :)

my warm wishes to u :)

btw do u have time to modify robots list...hence theres no one taking care of robots list... ?

[993ti]

Omg, look at Gigabot and Yahoo
http://www.twenteracingscene.com/spiders.php
Is that normal?

[mikelbeck]

Omg, look at Gigabot and Yahoo
http://www.twenteracingscene.com/spiders.php
Is that normal?
Yeah, they do a lot of that. Have a look at what they've done to one of my sites: http://www.happyhourpub.com/spiders.php.

Check out MSNbot, too, over 37K hits.

[Megareus Rex]

Uhh, a weird occurence:

When I first clicked on it a few minutes ago (after upgrading yesterday), it said Yahoo! Slurp had 390 IPs. I clicked on the link to display them, and it didn't open. I went and re-accessed the page, and it was down to 22. Now, every time I access spiders.php, the # of IPs for Y! Slurp goes down by 1.

Just an odd occurence...

PS: http://www.evermoreforums.com/forums/spiders.php if you wanna see

[Brandon Sheley]

upgraded and all seems fine..

thx for the update :)

btw, I like the popup for the ip's..
much cleaner..

[mikelbeck]

Uhh, a weird occurence:

When I first clicked on it a few minutes ago (after upgrading yesterday), it said Yahoo! Slurp had 390 IPs. I clicked on the link to display them, and it didn't open. I went and re-accessed the page, and it was down to 22. Now, every time I access spiders.php, the # of IPs for Y! Slurp goes down by 1.

Just an odd occurence...

PS: http://www.evermoreforums.com/forums/spiders.php if you wanna see
Yeah, I see that happening too.

It'll be fixed in the next version...

[funinthesun]

I've just installed this and got this error:
Database error in vBulletin 3.5.3:

Invalid SQL:
START TRANSACTION;

MySQL Error : You have an error in your SQL syntax near 'START TRANSACTION' at line 1
Error Number : 1064
Date : Friday, February 24th 2006 @ 06:33:48 PM
Script : http://www.********.com/spiders.php
Referrer :
IP Address : **.**.***.***
Username : *****
Classname : vb_database
I've also just read through 16 pages on this thread so not to bother you if the problem had already been solved with someone else but the two times that it's mentioned nobody has replied with a solution. So what I'm asking nicely is... can somebody help me please :nervous: :D

[Robbed]

I just got the same error, any fix?

[Robbed]

I figured it out, It's most likely because we are using sql version lower then 4.0. Start Transaction is only used in 4 and higher.

I opened up spiders.php line 47

replaced


$cleanup_data = $db->query_first("START TRANSACTION");

with

$cleanup_data = $db->query_first("BEGIN");

so far it's working.

[funinthesun]

Thanks! Mine is working now aswell and yes I'm running MySQL3 so looks like we have a fix ;)

[mikelbeck]

I figured it out, It's most likely because we are using sql version lower then 4.0. Start Transaction is only used in 4 and higher.

I opened up spiders.php line 47

replaced




with



so far it's working.

What does the "BEGIN" command do? Is it the same as "START TRANSACTION" in later versions?

[Robbed]

What does the "BEGIN" command do? Is it the same as "START TRANSACTION" in later versions?

ya, i was looking at what was new in 4.0

Added START TRANSACTION (SQL-99 syntax) as alias for BEGIN. This is recommended to use instead of BEGIN to start a transaction.

[mikelbeck]

ya, i was looking at what was new in 4.0
Ah, good. Thanks for that info, I'll add a check to see what version of MySQL it is and will use "BEGIN" for < 4 and "START TRANSACTION" for 4 and above in the next version.

[Zia]

helo mikelbeck..
hope ur going to release a new version.....
currently the page showing known & un-known spider

i think it can devided in few more portion

1.Seachengine Spider
2.Blog/Validator [Html,css,rss,xhtml,ror -all kinds of validator]
3.Feed Finder [all kinds of rss/xml feed finder or updater]
4.Misc or Unknown

currently the page showing the url bellow the bots nick...it would be nice if the <info> [robotstxt.org's url] line of the list.xml showed bellow the nick nick

after all of that..i would like to thankx u A LOT...cos w/o this i cant never relaise..which/how much spider we get in our site......

Thnx

[mikelbeck]

Take a look, let me know if this is what you're talking about: http://www.happyhourpub.com/spiders.php

[Zia]

yeah all most right...
i gez u catagorise spider in happyhourpub : Search Spider , http_monitoring,linkchecker,unknown
- Its not showing its Ip (Clickable details).
-Unknown spider block: ,spider name mentioned Unknown Spider Ipselonbot/1.14-beta (+http://www.ipselon.com/intl/en/ipselonbot.html)

Could it be devided: Searchspider,http_monitoring,Linkchecker,Rss/Blog Spider/Bot,Misc(Misc inclided unknown ,but in the Spider name "Unknown Spider" should not be mentioned.As much as info possible show.
- IP address(klikable list)

in every block every spider/bots as much as info possible to show.

umm but could u say..with this hack there is no relation with spiderlist.xml ?
im asking coz.some spider i found that they arent listed in the spiderlist.xml.
im not that xpert...could u plz tell me?

one more thing is there any way during upgrade previous spiders details (hit,time) all lost, how to prevent it?

Thnx.

[mikelbeck]

yeah all most right...
i gez u catagorise spider in happyhourpub : Search Spider , http_monitoring,linkchecker,unknown
- Its not showing its Ip (Clickable details).

IPs are only shown to users with permission to see IPs, which are admins in most cases.


-Unknown spider block: ,spider name mentioned Unknown Spider Ipselonbot/1.14-beta (+http://www.ipselon.com/intl/en/ipselonbot.html)

If the item is listed in the spiders_vbulletin.xml file as "unknown", it's going to be grouped with the "unknown" spiders. If the spider presents a user agent (like this one did), it'll be shown.


Could it be devided: Searchspider,http_monitoring,Linkchecker,Rss/Blog Spider/Bot,Misc(Misc inclided unknown ,but in the Spider name "Unknown Spider" should not be mentioned.As much as info possible show.


Only if that information is in the spiders_vbulletin.xml file, which it isn't.


umm but could u say..with this hack there is no relation with spiderlist.xml ?
im asking coz.some spider i found that they arent listed in the spiderlist.xml.
im not that xpert...could u plz tell me?


It gets it's information from the vbulletin_spiders.xml file (the one in includes/xml).

one more thing is there any way during upgrade previous spiders details (hit,time) all lost, how to prevent it?

Thnx.

That only occurs if there is a database change from the previous version. So far, the one I'm working on now (beta 9) has no databases changes so that data won't be lost.

[Zia]

that means..i must modify my vbulletin_spiders.xml to grouped/devided all the spider ?
i am curious there are some rss discovery/blog spider arent listed in the vbulletin_spiders.xml . I download the latest vbulletin_spiders.xml from vb.com
therad :http://www.vbulletin.com/forum/showthread.php?t=76662&highlight=spider
download:http://www.vbulletin.com/forum/showpost.php?p=565415&postcount=12

even there arent listed...


i also collect some spider url from urs spiderwatcher :D(plz dont remove detilas info)


Thankx waiting for new release.

[mikelbeck]

that means..i must modify my vbulletin_spiders.xml to grouped/devided all the spider ?
i am curious there are some rss discovery/blog spider arent listed in the vbulletin_spiders.xml . I download the latest vbulletin_spiders.xml from vb.com
therad :http://www.vbulletin.com/forum/showthread.php?t=76662&highlight=spider
download:http://www.vbulletin.com/forum/showpost.php?p=565415&postcount=12

even there arent listed...


Yes, you can just add them to the end of that file, and then they'll be displayed properly in the spider listing.

Thankx waiting for new release.

A few days, I've got one last issue to work out then I'll put it out. This may be the final release... I'll leave it in Beta for a week or so, as long as there's no issues I'll put it out as a final release.

[Zia]

:) getting an final release is always nice ..
i think this addon's..modification will never be end ;)

can u give a thought to reduce sql-query? in our sites list page (http://www.golpo.net/forum/spiders.php) it giving 268 query...
any way to reduce the query?

Page generated in 1.38554 seconds with 268 queries


errrrrrrrrr....one more thing..maybe its out of the topic.....but cant stop my- self :nervous:

asking about RSS, hence current Show Spider On Forum Home (https://vborg.vbsupport.ru/showthread.php?t=91569&highlight=spider+forum+home) hack dont work on portal home..if rss works on only the spiders name..that visited (today or all over) can be shown on portal home .if u are willing to do THIS Rrss icon in each forum (https://vborg.vbsupport.ru/showthread.php?t=104978&highlight=show+rss+each+forum)
might provide some idea...

if possible plz give a thought on it....

thanx

[SIINSI]

I installed this a few weeks ago and it worked fine. Now when I try to access the page, it just hangs... any clues?

Thanks.

[mikelbeck]

I installed this a few weeks ago and it worked fine. Now when I try to access the page, it just hangs... any clues?

Thanks.
What version do you have installed? Is it an active site, with a lot of spider activity?

[SIINSI]

What version do you have installed? Is it an active site, with a lot of spider activity?


Looks as though I am using Beta 7. My site is very active with alot of Spider activity. Does this mean the script won't do well for me?

Thanks for the quick response! :)

[mikelbeck]

Looks as though I am using Beta 7. My site is very active with alot of Spider activity. Does this mean the script won't do well for me?

Thanks for the quick response! :)

If you've got a lot of spider activity, it's going to take a little longer to collate it and display it. Once that's done, however, the page should load quicker.

The next version has some more reduction in the number of queries and should help with this situation.

[SIINSI]

I let the page sit for an hour and it didn't do anything but, cause alot of server load :)

[mikelbeck]

Version 1.0.0 Beta 9 is now available.

1.0.0 Beta 9 - 03/10/2006
- Changed the display to group similar spiders together (search spiders, http check spiders, etc)

[Mudvayne]

Updated :).. !seen Zia :P.. Newayz thnx Mikelbeck..

[XtremeOffroad]

thanx fot the update, argh all the activity is gone now. but thanks again

[nitro]

Updated seems ok :)

Think I would prefer a popup window that can be used like being able to copy the IP this would make it easier to add the bots ip to a banned IP list if you didnt want it on your site and it was ignoring robot.txt.

Another little suggestion the Bot name could be a link to a search engine so you can quickly access links about a bot and its origins and behaviour.