vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vBulletin 3.5 Add-ons (https://vborg.vbsupport.ru/forumdisplay.php?f=113)
-   -   Spider Watcher (https://vborg.vbsupport.ru/showthread.php?t=104582)

XtremeOffroad 01-25-2006 01:48 AM

Getting this error
Warning: array_multisort(): Argument #1 is expected to be an array or a sort flag in /spiders.php on line 223

And still no spiders show on my site.
I had asked for help with this in the past with no reply.

Zia 01-25-2006 02:19 AM

sounds nice.....
this detect search sipder & unknown spider.....but how to show the bot's home page url in the bots list page(generated page) bellow the botnick. ?

Just curious any one tried to detect unknown spiders??

thnx

Megareus Rex 01-25-2006 02:24 AM

A concern and suggestion.

One of the spiders (Yahoo! Slurp) is returning literally DOZENS of IPs. Here's the particular block I have now (and I've only had it for a couple days):

Quote:

0.0.0.115, 68.142.249.15, 68.142.249.19, 68.142.249.20, 68.142.249.21, 68.142.249.25, 68.142.249.31, 68.142.249.35, 68.142.249.38, 68.142.249.42, 68.142.249.44, 68.142.249.47, 68.142.249.48, 68.142.249.58, 68.142.249.61, 68.142.249.67, 68.142.249.68, 68.142.249.73, 68.142.249.81, 68.142.249.83, 68.142.249.84, 68.142.249.85, 68.142.249.91, 68.142.249.92, 68.142.249.96, 68.142.249.98, 68.142.249.99, 68.142.249.102, 68.142.249.108, 68.142.249.110, 68.142.249.112, 68.142.249.115, 68.142.249.116, 68.142.249.117, 68.142.249.119, 68.142.249.120, 68.142.249.124, 68.142.249.127, 68.142.249.132, 68.142.249.152, 68.142.249.154, 68.142.249.159, 68.142.249.164, 68.142.249.168, 68.142.249.176, 68.142.249.188, 68.142.249.191, 68.142.249.201, 68.142.249.207, 68.142.249.208, 68.142.250.11, 68.142.250.12, 68.142.250.13, 68.142.250.14, 68.142.250.15, 68.142.250.22, 68.142.250.26, 68.142.250.28, 68.142.250.36, 68.142.250.43, 68.142.250.53, 68.142.250.65, 68.142.250.73, 68.142.250.77, 68.142.250.79, 68.142.250.83, 68.142.250.86, 68.142.250.91, 68.142.250.93, 68.142.250.101, 68.142.250.102, 68.142.250.111, 68.142.250.114, 68.142.250.116, 68.142.250.118, 68.142.250.119, 68.142.250.122, 68.142.250.124, 68.142.250.126, 68.142.250.130, 68.142.250.131, 68.142.250.141, 68.142.250.142, 68.142.250.147, 68.142.250.152, 68.142.250.153, 68.142.250.154, 68.142.250.155, 68.142.250.158, 68.142.250.163, 68.142.250.165, 68.142.250.167, 68.142.250.169, 68.142.250.172, 68.142.250.176, 68.142.250.180, 68.142.250.181, 68.142.250.183, 68.142.250.187, 68.142.250.189, 68.142.250.193, 68.142.250.199, 68.142.250.202, 68.142.250.203, 68.142.250.208, 68.142.251.14, 68.142.251.18, 68.142.251.19, 68.142.251.23, 68.142.251.25, 68.142.251.34, 68.142.251.46, 68.142.251.47, 68.142.251.59, 68.142.251.69, 68.142.251.81, 68.142.251.85, 68.142.251.86, 68.142.251.92, 68.142.251.96, 68.142.251.101, 68.142.251.110, 68.142.251.113, 68.142.251.119, 68.142.251.123, 68.142.251.129, 68.142.251.132, 68.142.251.144, 68.142.251.153, 68.142.251.154, 68.142.251.155, 68.142.251.159, 68.142.251.167, 68.142.251.170, 68.142.251.180, 68.142.251.184, 68.142.251.185, 68.142.251.190, 68.142.251.196, 68.142.251.201, 68.142.251.203, 202.160.180.127
You might want to consider adding an IP display limit with a "View All" link to see all the IPs at your request, rather than displaying all on the main page itself.

mikelbeck 01-26-2006 12:35 AM

Chris-777: Still working on that. Don't worry, I haven't forgotten about you.

XtremeOffroad: That sounds like there's nothing to be sorted, so the multisort is failing. I will put a check in there in the next version. As for no spiders... Do they visit your site? Are you sure that the plug-in is installed and active?

Zia: What if I made the spider's name a hotlink instead of displaying the URL under it?

Megareus Rex: I'll see what I can do with that. Maybe just display 5 or so with a "view others" link to show the rest? Keep in mind that only users who have the "view IP" privliledge (such as admins or moderators) will see the IPs. All other users see nothing.

Everybody: What about the number of queries?

Guest210212002 01-26-2006 12:37 AM

:D Thanks dude. If it's just me that has the problem, no worries, I'll get by. It's a fantastic hack otherwise.

/salute

Megareus Rex 01-26-2006 12:43 AM

Quote:

Originally Posted by mikelbeck
Chris-777: Still working on that. Don't worry, I haven't forgotten about you.

XtremeOffroad: That sounds like there's nothing to be sorted, so the multisort is failing. I will put a check in there in the next version. As for no spiders... Do they visit your site? Are you sure that the plug-in is installed and active?

Zia: What if I made the spider's name a hotlink instead of displaying the URL under it?

Megareus Rex: I'll see what I can do with that. Maybe just display 5 or so with a "view others" link to show the rest? Keep in mind that only users who have the "view IP" privliledge (such as admins or moderators) will see the IPs. All other users see nothing.

Everybody: What about the number of queries?

A "View All" link would definitely be good. I know only those with IP viewing perms can see them, but I'm one of those people, and its rather annoying to see the Yahoo! Slurp spider growing ever larger... :P

As for the # of queries....well....just check this link out (look at the bottom):
http://www.evermoreforums.com/forums/spiders.php

I've been getting 800+ queries, though it seems to be growing (was only 700+ a few hours ago). So yeah...loads.

mikelbeck 01-26-2006 12:56 AM

Quote:

Originally Posted by Megareus Rex
I've been getting 800+ queries, though it seems to be growing (was only 700+ a few hours ago). So yeah...loads.

Yeah, that's way too much. I'll have to take another look at that.

I know how to fix it, the problem is I don't want to do it that way. What happens is this... When a guest arrives at a page, the spider plug in writes a line to the database. Just the timestamp, user agent and page. When you run the spiders page, it "rolls up" all of the data so it's displayed in a nice, neat format and all the old data is removed. That's what's creating so many queries.

If I were to change the plug in to check to see if there's already a record for that spider in the database and then just update it as the page is loaded, there would be no need to "roll up". But I think that would add a few queries on each page load (by a non-user, meaning a guest or spider) and I don't think that's a good idea.

I'll see what else I can come up with.

Megareus Rex 01-26-2006 02:52 AM

Quote:

Originally Posted by mikelbeck
Yeah, that's way too much. I'll have to take another look at that.

I know how to fix it, the problem is I don't want to do it that way. What happens is this... When a guest arrives at a page, the spider plug in writes a line to the database. Just the timestamp, user agent and page. When you run the spiders page, it "rolls up" all of the data so it's displayed in a nice, neat format and all the old data is removed. That's what's creating so many queries.

If I were to change the plug in to check to see if there's already a record for that spider in the database and then just update it as the page is loaded, there would be no need to "roll up". But I think that would add a few queries on each page load (by a non-user, meaning a guest or spider) and I don't think that's a good idea.

I'll see what else I can come up with.

Just letting you know, its almost at 1100 queries, and a page generation time of 7+ seconds.

Not to mention the Yahoo Slurp spider's # of IPs has more than doubled (perhaps close to tripled) from earlier.

Brandon Sheley 01-26-2006 07:15 AM

Quote:

Originally Posted by mikelbeck
Are the times & dates being displayed properly for you?
Have the number of queries decreased for you?


yes, queries are much lower.. under 100

not sure how to check time ? is there a way to reset,, so it shows zero spiders viewed ?

this would be handy i think..

so i could reset once a week, or a month,, to see how the spider traffic is...

thank you,, good hack still..

[high]* Brandon Sheley upgraded with no problems
[/high]

http://locoforum.com/forums/spiders.php
Page generated in 0.22789 seconds with 79 queries

Megareus Rex 01-26-2006 10:56 AM

And now my queries are up to 1750....


All times are GMT. The time now is 11:02 AM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01393 seconds
  • Memory Usage 1,757KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (5)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (4)pagenav_pagelink
  • (2)pagenav_pagelinkrel
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (10)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 

Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • pagenav_page
  • pagenav_complete
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete