Spider Watcher

Spider Watcher
Author: Mikel Beck (mikel.beck@elite-computing.net)

This hack keeps track of the spiders (Search Engine robots) that visit your fourm. Every time a guest visits a page, the guest's IP address, user agent and the page they visited are logged to the database.

When somebody views the spider statistics page, this data is "rolled up", meaning the raw data is collated, the spider's name is determined by comparing the user agent to data contained in the spiders_bulletin.xml file, and the number of pages and visits is summarized and writted back to the database. In addition, and data from non-bots is removed.

The data is then displayed in a easy to read format for your viewing pleasure.

If the user viewing the report has permissions to view IP addresses, these are displayed as well.

A live version of the report from one of my sites can be seen here: http://www.happyhourpub.com/spiders.php

Also see the attached screenshot for an exmaple.


Revision History:
1.0.0 Beta 1 - 01/05/2006
- Initial Release

1.0.0 Beta 2 - 01/06/2006
- Included templates for spiders.php
- Removed text from templates, added them as phrases

1.0.0 Beta 3 - 01/07/2006
- Split up the display of "known" and "unknown" spiders

1.0.0 Beta 4 - 01/25/2006
- Corrected potentional SQL injection issue in plug-in
- Reduced the number of SQL queries required to display statistics
- Corrected date/time display issue

1.0.0 Beta 5 - 02/01/2006
- Reduced the number of SQL queries required to display statistics

1.0.0 Beta 6 - 02/08/2006
- No release

1.0.0 Beta 7 - 02/11/2006
- Corrected issue with "unknown" spiders not being displayed properly.
- Added tracking of the type of spider (searchspider, link checker, etc)

1.0.0 Beta 8 - 02/19/2006
- Change the display of IP addresses to be a pop-up so they're all not displayed on the main page.
- Combined the spiders that have the same name but different user agents.

1.0.0 Beta 9 - 03/10/2006
- Changed the display to group similar spiders together (search spiders, http check spiders, etc)

1.0.0 Beta 10 - 08/08/2006
- Changed how the rollup functions. Instead of rolling up every time somebody views the spider page, it rolls up once per hour.
- Corrected a few bugs here and there, mostly related to removing entries from the database.

Installation Instructions
1. Upload spiders.php to the root of your forum.
2. Upload spiders_rollup.php to the includes/cron directory.
3. Import the file product-spiderwatcher.xml using the Manage Products module.
4. Add a link to spiders.php on your navbar or footer.
5. Add a cron job with the following information:
Title: Spider Watcher Rollup
Day of the Week: *
Day of the Month: *
Hour: *
Minute: 0 - - -
Log entries: Yes
Filename: ./includes/cron/spiders_rollup.php

Upgrade Instructions
1. Upload (and overwrite) spiders to the root of your forum.
2. Upload spiders_rollup.php to the includes/cron directory.
3. Import the file product-spiderwatcher.xml using the Manage Products module. Make sure the "Allow Overwrite" option is set to "Yes".
4. Add a link to spiders.php on your navbar or footer.
5. Add a cron job with the following information:
Title: Spider Watcher Rollup
Day of the Week: *
Day of the Month: *
Hour: *
Minute: 0 - - -
Log entries: Yes
Filename: ./includes/cron/spiders_rollup.php

***UPGRADE NOTE***
When you upgrade from version 1.0.0 Beta 7 to 1.0.0 Beta 8 your existing spider data will be lost!

To make sure that you can decode the maximum amount of spiders, you should grab the latest spiderlist.xml and replace the spiders_vbulletin.xml file in your forumhome/includes/xml/ directory with the one from this thread: http://www.vbulletin.com/forum/showthread.php?t=76662

[Totti]

Originally Posted by mikelbeck

How do you enable the page generation time for each page? Or is that a hack?

search for microstats on this board

[StarBuG]

Fatal error: Maximum execution time of 30 seconds exceeded in /path/forum/spiders.php on line 70

Guess it is not usable for me either but it is a nice idea.

Needs some improvements

[DementedMindz]

Originally Posted by mikelbeck

How do you enable the page generation time for each page? Or is that a hack?

here is the link for the plugin https://vborg.vbsupport.ru/showthread.php?t=82900

[Detomah]

I'm seriously thankful, yet at the same time wishing to god I hadn't installed this hack a few days ago.....

I've been monitoring some of the spiders that visit and i've got a bit of a shock through the addon.

Some of the visitors I thought were guests turn out to be rogue spiders, such as email address hunters, site grabbers and the likes.

I hadn't spotted them in the past, so never knew they were visiting, I just thought they were guests, as that's what it comes up as.

Anyway, after a bit of tweaking and after finding this site:

http://www.kloth.net/internet/badbots.php

Which displays a pretty good list of bad bots and even shows you a couple of good ways to stop them.

I'm now getting a much smoother image of who my visitors are and how many of them are actually real people.

So...

Thanks a bunch for this hack, I wouldn't have believed it would come in so handy.

The page is getting pretty big, pretty fast on my site though and I'm sure it's gonna start timing out soon.

Is there any way you could make it, so it runs over multiple pages instead or something like that, to try and cut down the queries?

[DementedMindz]

Detomah reading what you just posted and from reading what he says are good and bad bots is funny.... they say them bots are bad? well strange look at his robots.txt lol he only has one bot in there lol so they must not be too bad huh? http://www.kloth.net/robots.txt

[darksoulz]

Originally Posted by mikelbeck

Do you still have the same issue if you apply the mod in this post:

https://vborg.vbsupport.ru/showpost....5&postcount=98

?

hopefully this doesn't turn out too ugly...

mysql> select bot,timestamp from bots where bot like '%Find%';     
+-----------------------------------------------------------------+---------------------+
| bot                                                             | timestamp           |
+-----------------------------------------------------------------+---------------------+
| findlinks/1.1-a8 (+http://wortschatz.uni-leipzig.de/findlinks/) | 2006-01-20 13:06:27 |
+-----------------------------------------------------------------+---------------------+
1 row in set (0.02 sec)

but on the spiders.php page:

FindLinks 01-19-2006 12:06 PM

[Detomah]

Originally Posted by DementedMindz

Detomah reading what you just posted and from reading what he says are good and bad bots is funny.... they say them bots are bad? well strange look at his robots.txt lol he only has one bot in there lol so they must not be too bad huh? http://www.kloth.net/robots.txt

Yeah I had a look at that before and it amused me a little.

I think he may be simply failing to practice what he preaches.

Although he did say himself, some of the "bad bots" he mentioned aint actually that bad.

[nitro]

Originally Posted by Detomah

Yeah I had a look at that before and it amused me a little.

I think he may be simply failing to practice what he preaches.

Although he did say himself, some of the "bad bots" he mentioned aint actually that bad.

Nope you both failed to read. It is pointless using robots.txt to dissallow bots that simply ignore or missuse the robots.txt standard. The majority of the bots in that list have the code "M" or "N" which means to ignore or missuse robots.txt

Hence he has a bot trap to deal with them instead.

[Brandon Sheley]

any word on dropping the queries down ?

[mikelbeck]

Originally Posted by Loco.M

any word on dropping the queries down ?

Not yet, I haven't had a chance to get back to this. I will try to have a look at it this evening or tomorrow morning.