Spider Watcher

Spider Watcher
Author: Mikel Beck (mikel.beck@elite-computing.net)

This hack keeps track of the spiders (Search Engine robots) that visit your fourm. Every time a guest visits a page, the guest's IP address, user agent and the page they visited are logged to the database.

When somebody views the spider statistics page, this data is "rolled up", meaning the raw data is collated, the spider's name is determined by comparing the user agent to data contained in the spiders_bulletin.xml file, and the number of pages and visits is summarized and writted back to the database. In addition, and data from non-bots is removed.

The data is then displayed in a easy to read format for your viewing pleasure.

If the user viewing the report has permissions to view IP addresses, these are displayed as well.

A live version of the report from one of my sites can be seen here: http://www.happyhourpub.com/spiders.php

Also see the attached screenshot for an exmaple.


Revision History:
1.0.0 Beta 1 - 01/05/2006
- Initial Release

1.0.0 Beta 2 - 01/06/2006
- Included templates for spiders.php
- Removed text from templates, added them as phrases

1.0.0 Beta 3 - 01/07/2006
- Split up the display of "known" and "unknown" spiders

1.0.0 Beta 4 - 01/25/2006
- Corrected potentional SQL injection issue in plug-in
- Reduced the number of SQL queries required to display statistics
- Corrected date/time display issue

1.0.0 Beta 5 - 02/01/2006
- Reduced the number of SQL queries required to display statistics

1.0.0 Beta 6 - 02/08/2006
- No release

1.0.0 Beta 7 - 02/11/2006
- Corrected issue with "unknown" spiders not being displayed properly.
- Added tracking of the type of spider (searchspider, link checker, etc)

1.0.0 Beta 8 - 02/19/2006
- Change the display of IP addresses to be a pop-up so they're all not displayed on the main page.
- Combined the spiders that have the same name but different user agents.

1.0.0 Beta 9 - 03/10/2006
- Changed the display to group similar spiders together (search spiders, http check spiders, etc)

1.0.0 Beta 10 - 08/08/2006
- Changed how the rollup functions. Instead of rolling up every time somebody views the spider page, it rolls up once per hour.
- Corrected a few bugs here and there, mostly related to removing entries from the database.

Installation Instructions
1. Upload spiders.php to the root of your forum.
2. Upload spiders_rollup.php to the includes/cron directory.
3. Import the file product-spiderwatcher.xml using the Manage Products module.
4. Add a link to spiders.php on your navbar or footer.
5. Add a cron job with the following information:
Title: Spider Watcher Rollup
Day of the Week: *
Day of the Month: *
Hour: *
Minute: 0 - - -
Log entries: Yes
Filename: ./includes/cron/spiders_rollup.php

Upgrade Instructions
1. Upload (and overwrite) spiders to the root of your forum.
2. Upload spiders_rollup.php to the includes/cron directory.
3. Import the file product-spiderwatcher.xml using the Manage Products module. Make sure the "Allow Overwrite" option is set to "Yes".
4. Add a link to spiders.php on your navbar or footer.
5. Add a cron job with the following information:
Title: Spider Watcher Rollup
Day of the Week: *
Day of the Month: *
Hour: *
Minute: 0 - - -
Log entries: Yes
Filename: ./includes/cron/spiders_rollup.php

***UPGRADE NOTE***
When you upgrade from version 1.0.0 Beta 7 to 1.0.0 Beta 8 your existing spider data will be lost!

To make sure that you can decode the maximum amount of spiders, you should grab the latest spiderlist.xml and replace the spiders_vbulletin.xml file in your forumhome/includes/xml/ directory with the one from this thread: http://www.vbulletin.com/forum/showthread.php?t=76662

[Nathan2006]

Thank you mikelbeck

Really nice

Install

[Nathan2006]

Originally Posted by mikelbeck

To make sure that you can decode the maximum amount of spiders, you should grab the latest spiderlist.xml and replace the spiders_vbulletin.xml file in your forumhome/includes/xml/ directory with the one from this thread: http://www.vbulletin.com/forum/showthread.php?t=76662

How do I update the spider list?

I had a look on there but I dont know how to

On this post its from 2001
http://www.vbulletin.com/forum/showpost.php?p=565415

Thank you for any help

[Zia]

helo mikelbeck..

is there any update gonna release for vb3.6.0 ?

its an important module.

thanks

[4yBak]

is there any update gonna release for vb3.6.0 ?

its an important module.

this product compatible with vB 3.6 Beta 2

[ILTK]

Quick question:

I see the numbers going up for example on the googlebot entry, but the date is not allways updated.

Is this because you only update the date when it's a GET and not a HEAD request? or could this be a bug?

Love the hack btw.

[big dan]

Great plugin, installed without a hitch and works perfectly for me on 3.5.4

Thanks!

[libertylounge]

On my forum, it's not recognizing what bots are what.. how can I remedy this?

They are currently all listed as "Unknown Spider"

edit:

Ok, I downloaded that spiderlist.xml from vbulletin.com and replaced my spiders_vbulletin.xml with it.. is there a way to "update" the current entries so I'm able to see which bots are which easier, or will that never update?

[Aneurysm]

Thanks! easy install, works great.

[ILTK]

Had some new spider break the script today, looks like some of the code in the user_agent made the sql bomb, I have removed the script until this can be resolved, this isn't affecting my forums, just when you load the spiders page.

Database error in vBulletin 3.5.4:

Invalid SQL:
DELETE FROM spider_watcher WHERE bot = '<a href='http://www.netforex.org'> Forex Trading Network Organization </a> info@netforex.org';

MySQL Error  : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'http://www.netforex.org'> Forex Trading Network Organization </a> info@' at line 1
Error Number : 1064
Date         : Saturday, August 5th 2006 @ 11:43:32 AM
Script       : http://www.<snipped by iltk>
Referrer     : 
IP Address   : 83.*.*.*
Username     : <snip>
Classname    : vb_database

[mikelbeck]

I realized it's been a ridicously long time since I've updated this mod, but I did it today...

This version dramatically cuts down on the number of SQL queries that occur when somebody views the spiders page. Instead of rolling up the spider data at that point, a cron job rolls up the data once an hour.

Please note that the installation and upgrade instructions have changed, and a new file is included.

ILTK - the error you've just posted should be corrected in this latest version.

I don't have 3.6 installed on any of my sites yet, so I can't confirm it works with it. I don't see why it wouldn't, but I can't say that it does for sure.