Mysql version is 3.23.58, PHP version is 4.3.11 running on Linux.

(Not sure what you mean by which DB, though).

Edit: Thanks for the quick reply though. :D Looks like a killer hack, and I can't wait to get it working.

Also, I'm not sure if it matters, but there was no spiders.php at all in my forums root.

I meant "MySQL", you answered the question. ;-)

I'll have to check if MySQL 3 supports transactions, I have a feeling by the error you've gotten that it doesn't.

You have to put it there manually (ftp it to the directory).

Aah, awrighty. :)

[high]* Club3G anxiously overs over the Install button![/high]

You may want to find something else to do for a while, I'm not going to be able to have a look at it until later tonight or tomorrow. ;-)

But no-one else apart from admins can see the link on the left column. :)

/me is still poised over the install button, anxiously hoping that his DB error goes away

Soon... Some things have come up in my life recently that require my full attention, this is going to have to be on hold for a little while. But I promise I will get back to it soon and will have something that will work for you.

Mikelbeck, I just installed this, and it's great. Within minutes, the Yahoo slurp spidered my site. It's now 2023 hrs. And I installed it ten minutes ago.
Something wrong with the "hours" code, as was mentioned before. Hope this helps. Good luck with the "life issues" man, I know how that goes.

Thanks!
rich

just curious who got howmay hit & which spider?

No problem at all bro, I appreciate the effort. Take care of yourself.

Very odd.

Try this, see if it makes a difference...

Open up spiders.php, find this block of code:

Right before it, add this:

Then view your spiders page again and see if the dates are correct. If they are, then there's an issue with that block of code that I'll have to work out. If they're NOT, then it's something to do with MySQL.

In either case, post your results here.

Well, that made a big difference. Now it returns a date and time of last visit, such as "2006-01-16 03:59:17 6", instead of "x hours ago"

Just for your information, today, one day after I installed this (?), it went from "20 hours ago" to "19 hours ago". As if it's confusing the days for hours.

I'll keep an eye on this and see what happens tomw.

But is the "2006-01-16 03:59:17" that it's now displaying the correct time & date for your current time zone?

I am assuming so. The last spider that came through was at 2006-01-16 18:58:53, which was an 21 minutes ago. I do not know the exact time it actually visited me however.

Looks like you fixed it.

I would just like to chime in with the 'me too' on the problem of it showing Yesterday when it should say Today. I dont know if it might be interfering, but we also have this hack installed.

On another note, is there any easy way to get it to not log the unknown spiders? It seems to be picking up regular users that are using non-standard user-agent strings and filling up the database to the point of making us hit php's execution time limit when trying to view the page.

Hello..
here is a idea..
spider watcher now listing spider in two section
1.Search Spider
2.Unknown spider.

Els that there are Forum Spider....
like boardreader www.boardreader.com
Its been listed in unknown spider tho its listed in list.xml

any way that list will show in 3 section Seach spider,Forum Spider , Unknwon Spider..

while spider have its address in the list..so if it possble the spider's name is Klik Able ..

Thanks

No, not yet, but at least I know what's causing it...

For those who are having the problem with the dates & times... What is your setting for "Format for Time" and "Format for Date" in the "Date and Time Options" in your admin control panel? Also, what setting do you have for "Datestamp Display Option"?

I believe if you change these lines of code in spiders.php:

to be:

...this problem should go away. If somebody could give that a shot and let me know I'd appreciate it.

Just curious, what's the problem? Mine seems to be working correctly with that little fix from yesterday. http://www.russbo.org/vb/spiders.php?

rich

works on 3.5.2 patched to 3.5.3
[high]* Brandon Sheley clicks install
[/high]

http://locoforum.com/forums/spiders.php

I used it for a little while, but the list on the spiders page of "unknown" spiders was getting HUGE; even after installing the recommended spider list xml file. I must have had 100+ unknown spiders listed.

On ours,
Format for date - m-d-Y
Format for time - h:i A
Datestamp display option is Yesterday/Today, but like I said in my previous post, we have a hack installed that makes it user configurable.

I'm at work right now and dont have access to the machine to edit the php file. I will do that when I get home tonight and report back if someone doesn't beat me to it.


Edit: Just noticed something, The problem seems to only be while I'm logged in. If i log out and view the page as an unregistered user, it displays correctly. Another thing i just noticed while digging deeper, its not just the yesterday/today thing that's not correct. Everything is a day off while logged in. One of the spiders shows as 01-14-2006 03:24 PM on the page, but in the database the timestamp is 2006-01-15 16:24:51. My timezone setting on the board is an hour off the default which would explain the time difference, but I cant figure out why its one off on the day everywhere.

http://gprime.net/board/spiders.php

well how come you dont add the spiders to your spiders.xml file? then you wouldnt have that....

If they were seeing the same thing we are, quite a few of the 'Unknown Spider' entries aren't spiders at all. They are humans with browsers that have a useragent that isn't being recognized.

Do you still have the same issue if you apply the mod in this post:

https://vborg.vbsupport.ru/showpost....5&postcount=98

?

the only thing that i see so far is just the high sql queries... i think this wouldnt be to good on a large site in which spiders visit alot... is there any way to cut down on them? cause to be honest i only have my site in testing and its already using 70 queries with only 12 spiders that visited so far

just for a example on a board with alot of spiders visiting i was just looking at darksoulz post about there site http://gprime.net/board/spiders.php look at how many sql queries it has and its alot .... Page generated in 6.39825 seconds with 1909 queries

ouch, mine is up to 101 qeries.. I'll have to shut this down unless this is resolved :(

gprime is even more now:
Page generated in 16.68188 seconds with 4357 queries#
LOL!!!

i got 188 querries ... but i thought about leaving the link private and when the site is called just 12 times a week i can our server can handle this ... wouldn't be really great, though when this will be fixed!

Yeow... I just ran that page, it was 12,000 queries.

How do you enable the page generation time for each page? Or is that a hack?

That's way too much, lemme go back through the code and see what I can do with it!

search for microstats on this board

Fatal error: Maximum execution time of 30 seconds exceeded in /path/forum/spiders.php on line 70

Guess it is not usable for me either but it is a nice idea.

Needs some improvements ;)

here is the link for the plugin https://vborg.vbsupport.ru/showthread.php?t=82900

I'm seriously thankful, yet at the same time wishing to god I hadn't installed this hack a few days ago.....

I've been monitoring some of the spiders that visit and i've got a bit of a shock through the addon.

Some of the visitors I thought were guests turn out to be rogue spiders, such as email address hunters, site grabbers and the likes.

I hadn't spotted them in the past, so never knew they were visiting, I just thought they were guests, as that's what it comes up as.

Anyway, after a bit of tweaking and after finding this site:

http://www.kloth.net/internet/badbots.php

Which displays a pretty good list of bad bots and even shows you a couple of good ways to stop them.

I'm now getting a much smoother image of who my visitors are and how many of them are actually real people.

So...

Thanks a bunch for this hack, I wouldn't have believed it would come in so handy. :)

The page is getting pretty big, pretty fast on my site though and I'm sure it's gonna start timing out soon.

Is there any way you could make it, so it runs over multiple pages instead or something like that, to try and cut down the queries?

Detomah reading what you just posted and from reading what he says are good and bad bots is funny.... they say them bots are bad? well strange look at his robots.txt lol he only has one bot in there lol so they must not be too bad huh? http://www.kloth.net/robots.txt

hopefully this doesn't turn out too ugly...

but on the spiders.php page:

FindLinks 01-19-2006 12:06 PM

Yeah I had a look at that before and it amused me a little.

I think he may be simply failing to practice what he preaches.

Although he did say himself, some of the "bad bots" he mentioned aint actually that bad.

Nope you both failed to read. It is pointless using robots.txt to dissallow bots that simply ignore or missuse the robots.txt standard. The majority of the bots in that list have the code "M" or "N" which means to ignore or missuse robots.txt

Hence he has a bot trap to deal with them instead. ;)

any word on dropping the queries down ?

Not yet, I haven't had a chance to get back to this. I will try to have a look at it this evening or tomorrow morning.