Spider Watcher

Spider Watcher
Author: Mikel Beck (mikel.beck@elite-computing.net)

This hack keeps track of the spiders (Search Engine robots) that visit your fourm. Every time a guest visits a page, the guest's IP address, user agent and the page they visited are logged to the database.

When somebody views the spider statistics page, this data is "rolled up", meaning the raw data is collated, the spider's name is determined by comparing the user agent to data contained in the spiders_bulletin.xml file, and the number of pages and visits is summarized and writted back to the database. In addition, and data from non-bots is removed.

The data is then displayed in a easy to read format for your viewing pleasure.

If the user viewing the report has permissions to view IP addresses, these are displayed as well.

A live version of the report from one of my sites can be seen here: http://www.happyhourpub.com/spiders.php

Also see the attached screenshot for an exmaple.


Revision History:
1.0.0 Beta 1 - 01/05/2006
- Initial Release

1.0.0 Beta 2 - 01/06/2006
- Included templates for spiders.php
- Removed text from templates, added them as phrases

1.0.0 Beta 3 - 01/07/2006
- Split up the display of "known" and "unknown" spiders

1.0.0 Beta 4 - 01/25/2006
- Corrected potentional SQL injection issue in plug-in
- Reduced the number of SQL queries required to display statistics
- Corrected date/time display issue

1.0.0 Beta 5 - 02/01/2006
- Reduced the number of SQL queries required to display statistics

1.0.0 Beta 6 - 02/08/2006
- No release

1.0.0 Beta 7 - 02/11/2006
- Corrected issue with "unknown" spiders not being displayed properly.
- Added tracking of the type of spider (searchspider, link checker, etc)

1.0.0 Beta 8 - 02/19/2006
- Change the display of IP addresses to be a pop-up so they're all not displayed on the main page.
- Combined the spiders that have the same name but different user agents.

1.0.0 Beta 9 - 03/10/2006
- Changed the display to group similar spiders together (search spiders, http check spiders, etc)

1.0.0 Beta 10 - 08/08/2006
- Changed how the rollup functions. Instead of rolling up every time somebody views the spider page, it rolls up once per hour.
- Corrected a few bugs here and there, mostly related to removing entries from the database.

Installation Instructions
1. Upload spiders.php to the root of your forum.
2. Upload spiders_rollup.php to the includes/cron directory.
3. Import the file product-spiderwatcher.xml using the Manage Products module.
4. Add a link to spiders.php on your navbar or footer.
5. Add a cron job with the following information:
Title: Spider Watcher Rollup
Day of the Week: *
Day of the Month: *
Hour: *
Minute: 0 - - -
Log entries: Yes
Filename: ./includes/cron/spiders_rollup.php

Upgrade Instructions
1. Upload (and overwrite) spiders to the root of your forum.
2. Upload spiders_rollup.php to the includes/cron directory.
3. Import the file product-spiderwatcher.xml using the Manage Products module. Make sure the "Allow Overwrite" option is set to "Yes".
4. Add a link to spiders.php on your navbar or footer.
5. Add a cron job with the following information:
Title: Spider Watcher Rollup
Day of the Week: *
Day of the Month: *
Hour: *
Minute: 0 - - -
Log entries: Yes
Filename: ./includes/cron/spiders_rollup.php

***UPGRADE NOTE***
When you upgrade from version 1.0.0 Beta 7 to 1.0.0 Beta 8 your existing spider data will be lost!

To make sure that you can decode the maximum amount of spiders, you should grab the latest spiderlist.xml and replace the spiders_vbulletin.xml file in your forumhome/includes/xml/ directory with the one from this thread: http://www.vbulletin.com/forum/showthread.php?t=76662

[MPDev]

I used it for a little while, but the list on the spiders page of "unknown" spiders was getting HUGE; even after installing the recommended spider list xml file. I must have had 100+ unknown spiders listed.

[darksoulz]

Originally Posted by mikelbeck

For those who are having the problem with the dates & times... What is your setting for "Format for Time" and "Format for Date" in the "Date and Time Options" in your admin control panel? Also, what setting do you have for "Datestamp Display Option"?

On ours,
Format for date - m-d-Y
Format for time - h:i A
Datestamp display option is Yesterday/Today, but like I said in my previous post, we have a hack installed that makes it user configurable.

I'm at work right now and dont have access to the machine to edit the php file. I will do that when I get home tonight and report back if someone doesn't beat me to it.


Edit: Just noticed something, The problem seems to only be while I'm logged in. If i log out and view the page as an unregistered user, it displays correctly. Another thing i just noticed while digging deeper, its not just the yesterday/today thing that's not correct. Everything is a day off while logged in. One of the spiders shows as 01-14-2006 03:24 PM on the page, but in the database the timestamp is 2006-01-15 16:24:51. My timezone setting on the board is an hour off the default which would explain the time difference, but I cant figure out why its one off on the day everywhere.

http://gprime.net/board/spiders.php

[DementedMindz]

Originally Posted by MPDev

I used it for a little while, but the list on the spiders page of "unknown" spiders was getting HUGE; even after installing the recommended spider list xml file. I must have had 100+ unknown spiders listed.

well how come you dont add the spiders to your spiders.xml file? then you wouldnt have that....

[darksoulz]

If they were seeing the same thing we are, quite a few of the 'Unknown Spider' entries aren't spiders at all. They are humans with browsers that have a useragent that isn't being recognized.

[mikelbeck]

Originally Posted by darksoulz

Edit: Just noticed something, The problem seems to only be while I'm logged in. If i log out and view the page as an unregistered user, it displays correctly. Another thing i just noticed while digging deeper, its not just the yesterday/today thing that's not correct. Everything is a day off while logged in. One of the spiders shows as 01-14-2006 03:24 PM on the page, but in the database the timestamp is 2006-01-15 16:24:51. My timezone setting on the board is an hour off the default which would explain the time difference, but I cant figure out why its one off on the day everywhere.

http://gprime.net/board/spiders.php

Do you still have the same issue if you apply the mod in this post:

https://vborg.vbsupport.ru/showpost....5&postcount=98

?

[DementedMindz]

the only thing that i see so far is just the high sql queries... i think this wouldnt be to good on a large site in which spiders visit alot... is there any way to cut down on them? cause to be honest i only have my site in testing and its already using 70 queries with only 12 spiders that visited so far

[DementedMindz]

just for a example on a board with alot of spiders visiting i was just looking at darksoulz post about there site http://gprime.net/board/spiders.php look at how many sql queries it has and its alot .... Page generated in 6.39825 seconds with 1909 queries

[Brandon Sheley]

ouch, mine is up to 101 qeries.. I'll have to shut this down unless this is resolved

[Totti]

gprime is even more now:
Page generated in 16.68188 seconds with 4357 queries#
LOL!!!

i got 188 querries ... but i thought about leaving the link private and when the site is called just 12 times a week i can our server can handle this ... wouldn't be really great, though when this will be fixed!

[mikelbeck]

Originally Posted by DementedMindz

just for a example on a board with alot of spiders visiting i was just looking at darksoulz post about there site http://gprime.net/board/spiders.php look at how many sql queries it has and its alot .... Page generated in 6.39825 seconds with 1909 queries

Yeow... I just ran that page, it was 12,000 queries.

How do you enable the page generation time for each page? Or is that a hack?

That's way too much, lemme go back through the code and see what I can do with it!