The Arcive of Official vBulletin Modifications Site.It is not a VB3 engine, just a parsed copy! |
|
|
|||||||
Spider Watcher Details »»
|
|||||||||||||||||||||||||
Spider Watcher
Author: Mikel Beck (mikel.beck@elite-computing.net) This hack keeps track of the spiders (Search Engine robots) that visit your fourm. Every time a guest visits a page, the guest's IP address, user agent and the page they visited are logged to the database. When somebody views the spider statistics page, this data is "rolled up", meaning the raw data is collated, the spider's name is determined by comparing the user agent to data contained in the spiders_bulletin.xml file, and the number of pages and visits is summarized and writted back to the database. In addition, and data from non-bots is removed. The data is then displayed in a easy to read format for your viewing pleasure. If the user viewing the report has permissions to view IP addresses, these are displayed as well. A live version of the report from one of my sites can be seen here: http://www.happyhourpub.com/spiders.php Also see the attached screenshot for an exmaple. Revision History: 1.0.0 Beta 1 - 01/05/2006 - Initial Release 1.0.0 Beta 2 - 01/06/2006 - Included templates for spiders.php - Removed text from templates, added them as phrases 1.0.0 Beta 3 - 01/07/2006 - Split up the display of "known" and "unknown" spiders 1.0.0 Beta 4 - 01/25/2006 - Corrected potentional SQL injection issue in plug-in - Reduced the number of SQL queries required to display statistics - Corrected date/time display issue 1.0.0 Beta 5 - 02/01/2006 - Reduced the number of SQL queries required to display statistics 1.0.0 Beta 6 - 02/08/2006 - No release 1.0.0 Beta 7 - 02/11/2006 - Corrected issue with "unknown" spiders not being displayed properly. - Added tracking of the type of spider (searchspider, link checker, etc) 1.0.0 Beta 8 - 02/19/2006 - Change the display of IP addresses to be a pop-up so they're all not displayed on the main page. - Combined the spiders that have the same name but different user agents. 1.0.0 Beta 9 - 03/10/2006 - Changed the display to group similar spiders together (search spiders, http check spiders, etc) 1.0.0 Beta 10 - 08/08/2006 - Changed how the rollup functions. Instead of rolling up every time somebody views the spider page, it rolls up once per hour. - Corrected a few bugs here and there, mostly related to removing entries from the database. Installation Instructions 1. Upload spiders.php to the root of your forum. 2. Upload spiders_rollup.php to the includes/cron directory. 3. Import the file product-spiderwatcher.xml using the Manage Products module. 4. Add a link to spiders.php on your navbar or footer. 5. Add a cron job with the following information: Title: Spider Watcher Rollup Day of the Week: * Day of the Month: * Hour: * Minute: 0 - - - Log entries: Yes Filename: ./includes/cron/spiders_rollup.php Upgrade Instructions 1. Upload (and overwrite) spiders to the root of your forum. 2. Upload spiders_rollup.php to the includes/cron directory. 3. Import the file product-spiderwatcher.xml using the Manage Products module. Make sure the "Allow Overwrite" option is set to "Yes". 4. Add a link to spiders.php on your navbar or footer. 5. Add a cron job with the following information: Title: Spider Watcher Rollup Day of the Week: * Day of the Month: * Hour: * Minute: 0 - - - Log entries: Yes Filename: ./includes/cron/spiders_rollup.php ***UPGRADE NOTE*** When you upgrade from version 1.0.0 Beta 7 to 1.0.0 Beta 8 your existing spider data will be lost! To make sure that you can decode the maximum amount of spiders, you should grab the latest spiderlist.xml and replace the spiders_vbulletin.xml file in your forumhome/includes/xml/ directory with the one from this thread: http://www.vbulletin.com/forum/showthread.php?t=76662 Supporters / CoAuthors Show Your Support
|
|||||||||||||||||||||||||
About Developer
Visit Web Site
DB Changes 
Is in Beta Stage | Comments |
|
#132
01-25-2006, 01:48 AM
|
|||
|
|||
Getting this error
Warning: array_multisort(): Argument #1 is expected to be an array or a sort flag in /spiders.php on line 223 And still no spiders show on my site. I had asked for help with this in the past with no reply. 
|
|
#133
01-25-2006, 02:19 AM
|
||||
|
||||
|
sounds nice.....
this detect search sipder & unknown spider.....but how to show the bot's home page url in the bots list page(generated page) bellow the botnick. ? Just curious any one tried to detect unknown spiders?? thnx
|
|
#134
01-25-2006, 02:24 AM
|
||||
|
||||
|
A concern and suggestion.
One of the spiders (Yahoo! Slurp) is returning literally DOZENS of IPs. Here's the particular block I have now (and I've only had it for a couple days): Quote:
|
|
#135
01-26-2006, 12:35 AM
|
||||
|
||||
|
Chris-777: Still working on that. Don't worry, I haven't forgotten about you.
XtremeOffroad: That sounds like there's nothing to be sorted, so the multisort is failing. I will put a check in there in the next version. As for no spiders... Do they visit your site? Are you sure that the plug-in is installed and active? Zia: What if I made the spider's name a hotlink instead of displaying the URL under it? Megareus Rex: I'll see what I can do with that. Maybe just display 5 or so with a "view others" link to show the rest? Keep in mind that only users who have the "view IP" privliledge (such as admins or moderators) will see the IPs. All other users see nothing. Everybody: What about the number of queries?
|
|
#136
01-26-2006, 12:37 AM
|
|||
|
|||
 Thanks dude. If it's just me that has the problem, no worries, I'll get by. It's a fantastic hack otherwise./salute
|
|
#137
01-26-2006, 12:43 AM
|
||||
|
||||
|
Quote:
As for the # of queries....well....just check this link out (look at the bottom): http://www.evermoreforums.com/forums/spiders.php I've been getting 800+ queries, though it seems to be growing (was only 700+ a few hours ago). So yeah...loads.
|
|
#138
01-26-2006, 12:56 AM
|
||||
|
||||
|
Quote:
I know how to fix it, the problem is I don't want to do it that way. What happens is this... When a guest arrives at a page, the spider plug in writes a line to the database. Just the timestamp, user agent and page. When you run the spiders page, it "rolls up" all of the data so it's displayed in a nice, neat format and all the old data is removed. That's what's creating so many queries. If I were to change the plug in to check to see if there's already a record for that spider in the database and then just update it as the page is loaded, there would be no need to "roll up". But I think that would add a few queries on each page load (by a non-user, meaning a guest or spider) and I don't think that's a good idea. I'll see what else I can come up with.
|
|
#139
01-26-2006, 02:52 AM
|
||||
|
||||
|
Quote:
Not to mention the Yahoo Slurp spider's # of IPs has more than doubled (perhaps close to tripled) from earlier.
|
|
#140
01-26-2006, 07:15 AM
|
||||
|
||||
|
Quote:
yes, queries are much lower.. under 100 not sure how to check time ? is there a way to reset,, so it shows zero spiders viewed ? this would be handy i think.. so i could reset once a week, or a month,, to see how the spider traffic is... thank you,, good hack still.. [high]* Brandon Sheley upgraded with no problems [/high] http://locoforum.com/forums/spiders.php Page generated in 0.22789 seconds with 79 queries
|
|
#141
01-26-2006, 10:56 AM
|
||||
|
||||
|
And now my queries are up to 1750....
|
 |
«
Previous Thread
|
Next Thread
»
|
|
All times are GMT. The time now is 06:26 AM.