Spider Watcher

Spider Watcher
Author: Mikel Beck (mikel.beck@elite-computing.net)

This hack keeps track of the spiders (Search Engine robots) that visit your fourm. Every time a guest visits a page, the guest's IP address, user agent and the page they visited are logged to the database.

When somebody views the spider statistics page, this data is "rolled up", meaning the raw data is collated, the spider's name is determined by comparing the user agent to data contained in the spiders_bulletin.xml file, and the number of pages and visits is summarized and writted back to the database. In addition, and data from non-bots is removed.

The data is then displayed in a easy to read format for your viewing pleasure.

If the user viewing the report has permissions to view IP addresses, these are displayed as well.

A live version of the report from one of my sites can be seen here: http://www.happyhourpub.com/spiders.php

Also see the attached screenshot for an exmaple.


Revision History:
1.0.0 Beta 1 - 01/05/2006
- Initial Release

1.0.0 Beta 2 - 01/06/2006
- Included templates for spiders.php
- Removed text from templates, added them as phrases

1.0.0 Beta 3 - 01/07/2006
- Split up the display of "known" and "unknown" spiders

1.0.0 Beta 4 - 01/25/2006
- Corrected potentional SQL injection issue in plug-in
- Reduced the number of SQL queries required to display statistics
- Corrected date/time display issue

1.0.0 Beta 5 - 02/01/2006
- Reduced the number of SQL queries required to display statistics

1.0.0 Beta 6 - 02/08/2006
- No release

1.0.0 Beta 7 - 02/11/2006
- Corrected issue with "unknown" spiders not being displayed properly.
- Added tracking of the type of spider (searchspider, link checker, etc)

1.0.0 Beta 8 - 02/19/2006
- Change the display of IP addresses to be a pop-up so they're all not displayed on the main page.
- Combined the spiders that have the same name but different user agents.

1.0.0 Beta 9 - 03/10/2006
- Changed the display to group similar spiders together (search spiders, http check spiders, etc)

1.0.0 Beta 10 - 08/08/2006
- Changed how the rollup functions. Instead of rolling up every time somebody views the spider page, it rolls up once per hour.
- Corrected a few bugs here and there, mostly related to removing entries from the database.

Installation Instructions
1. Upload spiders.php to the root of your forum.
2. Upload spiders_rollup.php to the includes/cron directory.
3. Import the file product-spiderwatcher.xml using the Manage Products module.
4. Add a link to spiders.php on your navbar or footer.
5. Add a cron job with the following information:
Title: Spider Watcher Rollup
Day of the Week: *
Day of the Month: *
Hour: *
Minute: 0 - - -
Log entries: Yes
Filename: ./includes/cron/spiders_rollup.php

Upgrade Instructions
1. Upload (and overwrite) spiders to the root of your forum.
2. Upload spiders_rollup.php to the includes/cron directory.
3. Import the file product-spiderwatcher.xml using the Manage Products module. Make sure the "Allow Overwrite" option is set to "Yes".
4. Add a link to spiders.php on your navbar or footer.
5. Add a cron job with the following information:
Title: Spider Watcher Rollup
Day of the Week: *
Day of the Month: *
Hour: *
Minute: 0 - - -
Log entries: Yes
Filename: ./includes/cron/spiders_rollup.php

***UPGRADE NOTE***
When you upgrade from version 1.0.0 Beta 7 to 1.0.0 Beta 8 your existing spider data will be lost!

To make sure that you can decode the maximum amount of spiders, you should grab the latest spiderlist.xml and replace the spiders_vbulletin.xml file in your forumhome/includes/xml/ directory with the one from this thread: http://www.vbulletin.com/forum/showthread.php?t=76662

[oberheimhaven]

Can someone throw me the code to add this to my navbar please I know Im learning thank for the great hack installed 3.5.4 no issues at all Just need quick code to add to nav bar

thxs
Mark

[oberheimhaven]

I got it I added to tool bar fav I dont need this inside my actually forums works great thanks

Markus

[Snake]

Originally Posted by mikelbeck

Ok, people, I've had some time to get back to this. Sorry it's taken so long....

I've got Beta 10 running here: http://www.heavymetalscene.com/spiders.php

Notice how much faster it is. I've pulled out the "rolling up" code and placed it in a file that's run once an hour via a cron job. So the data isn't 100% up to date, but it uses much (much) less queries than the previous version.

What else do you want to see in this release?

Oh that is great but have you updated the zip pack yet?

[mikelbeck]

Originally Posted by Dark Cloud

Oh that is great but have you updated the zip pack yet?

Not yet, give me a couple of days.

[Snake]

Alrighty then, I'll be waiting here for the update.

[jd_sa_bowler]

Hi guys,

like the mod, I have installed by I am unsure if it is working correctly.

mysite http://www.bowls-aust.com/spiders.php

The problem is no type display for spiders except unknown and the google spider seems to be in the correct address range to be identified by the spiders_vbulletin.xml file data.

I uploaded the latest spider file and rename to spiders_vbulletin.xml

Am I just to impatient or is something wrong.

[Stadler]

I don't know, if this is already suggested, but I suggest, that you add an option to only list spiders with at least X Hits in the list of unknown spiders and perhaps add a link to 'List all unknown spiders' then. especially these garbled user agents, like 'cblbshjhbjdmpvrbfdrvv5arh5wi' for example seem to me like some crappy configured firewall or proxy, which tries to hide the user-agent ...

[Svenna]

I straight one mySQL errors indicated got. Where does the problem lie?

Invalid SQL:
DELETE FROM spider_watcher WHERE bot = 'Netluchs/0.8-dev ( ; http://www.netluchs.de/; ___don't___spam_me_@netluchs.de)';

MySQL-Error : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 't___spam_me_@netluchs.de)'' at line 1

Any can help?

[4yBak]

Originally Posted by Svenna

Any can help?

what you do when this error going? Query like this may be possible SQL-injection did posible hackers, but I'm not sure about it, so don't worry early

mikelbeck
can you do display IP of spiders via popupable div? Because JScript popup don't usable

This plugin don't want add in list spider WebAlta. In table spider_watcher I see always random fileds like this:

56680 | WebAlta Crawler/1.3.15 (http://www.webalta.ru/bot.html) | Webalta Crawler | 2006-07-14 20:18:55 | 1373540995 | showthread | searchspider | http://www.webalta.ru/bot.html | 0
56681 | WebAlta Crawler/1.3.15 (http://www.webalta.ru/bot.html) | Webalta Crawler | 2006-07-14 20:19:42 | 1373540995 | showthread | searchspider | http://www.webalta.ru/bot.html | 0
...

So, as you see - script do not count rollup. Why?

[Snake]

No update yet?