List Search engine Spider at Who's online

[Stadler]

I've created this hack, cause I wanted to see, when search engine spiders are browsing my forums.

This hack adds a textbox, where you can enter a list of Spider-IPs, so they would be recognized as 'Search engine spider' on who's online.

I'm using the lists from www.iplists.com right now and it seems to work without problems, though I've had to add some IPs from Inktomisearch that aren't listed there.

First I've used parts of the banip-function, but that was really slow then, so I've had to tweak it. Now its only slightly slower, than without it. You won't even recognize it.

I've ported it to the banip-function in order to reduce server-load, when using a big list of banned IPs. I'll release that, when I'm finished here.

Screenshots:
Search engine spider browsing forums (https://vborg.vbsupport.ru/attachment.php?postid=405909)
New textbox to add IPs in Admin-CP (https://vborg.vbsupport.ru/attachment.php?postid=405911)

PS: I'd be happy, if you only click install, if you really installed it, since I want to know, how many really installed it and since I need feedback.

I've added a version, that uses the useragent rather, than using an IP-List. I recommend using spiderips_useragent.txt from now on.

[Stadler]

Search engine spider browsing forums

[Stadler]

New textbox to add IPs in Admin-CP

[Beanyhead]

* Beanyhead installs :)

[kaotic]

Nice hacks. I plan to install both of them tomorrow, as I don't have enough time right now. :(

[Stadler]

That would be good :)

I'm running it for some days on our forums now and have occurred no problems so far, but maybe there are still bugs I haven't occured yet.

[Dean C]

Only problem is this relies on knowing the IP's of the spiders and recognising them ;)

- miSt

[Stadler]

e. g. with the lists from www.iplists.com? ;)

[gmarik]

Great, I got the same idea a week ago ... and than I found it's done by you.

[MrNase]

Hey great hack, clicked install...

but howto? just copy&paste the ip's from iplist.com to acp ?

[Dean C]

06-08-03 at 03:10 PM Stadler said this in Post #8 (https://vborg.vbsupport.ru/showthread.php?postid=406094#post406094)
e. g. with the lists from www.iplists.com? ;)

Touch? ;)

[monstergamer]

cool hack
**clicks install**

[Freddie Bingham]

Maybe you want to look at the useragent instead of going by ip (the vb3 way)? It can be spoofed but it is a whole lot simpler and faster. Look at Who's Online on vbulletin.com

preg_match('#(googlebot|lycos|ask jeeves|scooter|fast-webcrawler|slurp@inktomi|turnitinbot)#'

[Kriek]

Test/Install via 2.3.0 failed, no errors, no change on who’s online.

[Stadler]

Unfortunately we're running vB3 now, so I need to install vB2.3.0 locally. Maybe I'll take a look in the next few days, to get it to work with vB2.3.0 and maybe I'll take a look at freddies suggestion, but I suppose I'll hardcode it then, like in vB3 Beta 3 so its easier to install.

[Blizzb]

Does this work for 2.3.0?

[Kriek]

Today at 02:51 PM Blizzb said this in Post #16 (https://vborg.vbsupport.ru/showthread.php?postid=419993#post419993)
Does this work for 2.3.0?
*sigh*

Blizzb, read my previous statement. :rolleyes:

Yesterday at 11:00 PM Kriek said this in Post #14 (https://vborg.vbsupport.ru/showthread.php?postid=419879#post419879)
Test/Install via 2.3.0 failed, no errors, no change on who?s online.

[Till]

Installation was pretty easy, good job. :)

I added the IPs from iplists.com, where do I see the active users though?

[PurpleCow]

Stadler - Its a nice hack, but can you get it to work on 2.3.0 stable release also ?

Thanks

[Stadler]

Sorry, but I can't find any problems. It worked without problems with a freash installation of vB 2.3.0 for me.

I'll take a look into freddies suggestion, soon. But I can only debug locally.

[Till]

Yesterday at 04:15 PM Stadler said this in Post #20 (https://vborg.vbsupport.ru/showthread.php?postid=422843#post422843)
Sorry, but I can't find any problems. It worked without problems with a freash installation of vB 2.3.0 for me.

I'll take a look into freddies suggestion, soon. But I can only debug locally.

Can you tell me where the screenshot is from where it showed the bot spidering in the list of active users?

[Stadler]

Who's Online? ... Click on 'Currently active Users: xx'

in german: 'Zur Zeit aktive Benutzer: xx' ;)

[Till]

Today at 12:47 AM Stadler said this in Post #22 (https://vborg.vbsupport.ru/showthread.php?postid=422938#post422938)
Who's Online? ... Click on 'Currently active Users: xx'

in german: 'Zur Zeit aktive Benutzer: xx' ;)

Oh! :D I disabled this one... hehe.

There is a crawler on my forum right now! Yay!!!

Danke. ;)

[Megadeuce]

This hack works well. I wish it could tell me who's spider it is but here is a screenshot....

Doesn't show up on who's online though

UPDATED: must switch on "must resolve IPs" in the admin control panels to see the name of the spider.

Can we get this to also show up on who visited the site today?

[Moyses]

this is a cool hack, doomo arigato

[Dominus]

vbulletin 2.3.2
------------------

Using freddies suggestion, works quite well... ;)

preg_match('#(googlebot|lycos|ask jeeves|scooter|fast-webcrawler|slurp@inktomi|turnitinbot)#'

See screenshot.

Unfortunately we're running vB3 now, so I need to install vB2.3.0 locally. Maybe I'll take a look in the next few days, to get it to work with vB2.3.0 and maybe I'll take a look at freddies suggestion, but I suppose I'll hardcode it then, like in vB3 Beta 3 so its easier to install.

[Stadler]

OK, now I've created a version, that works similar to freddies suggestion. But I need someone, who tests it for me. First, that I didn't forgot something in the attached file and second, that it works fine in a live environment.

***The Attachment was moved to the first post***

[Rampag33]

Where's the user-agent box to add ips

I ran the query but don't see it in the admin cp

search the database and don't find anything realted to the query for it.

[Dominus]

Where's the user-agent box to add ips

I ran the query but don't see it in the admin cp

search the database and don't find anything realted to the query for it.
You do not add IPS, you add useragents into this.

// spider list
$spidernamelist = array(
'scooter' => 'Altavista',
'fast-webcrawler' => 'AlltheWeb',
'ask jeeves' => 'Ask Jeeves',
'googlebot' => 'Google',
'grub.org' => 'Grub Client',
'slurp@inktomi' => 'Inktomi',
'lycos' => 'Lycos',
'turnitinbot' => 'Turnitin.com'
);

[Rampag33]

OK cool :) damn come on bots

*** installed ***

[Dimitrix]

Great hack,
works like a charm :)

So far only GoogleBots bother me but Ill keep my eyes open :)
Thanks again
Dimitry

[Highlander]

i have a problem with a new searchbot .. ..always when he is coming.. i have the error displayed on the screen and he is messig up the whole db .. i will loose the mysql server if i dont switch the forum offline .. he is making a server load of 7 or more .. normaly when i have 100 users online i have maybe a server load of 0.5 maximum 1.2 ..the sql server crashes when this bot is coming..

what i can do? deinstall this hack?

[Highlander]

in index.php on 219 it writes following code:

if (!(preg_match("/$webrobot/i", $loggedinrobots) )) $loggedinrobots .= $webrobot . ", ";


need help.. quickly please .. it crashes nearly every three ays my server when he comes for a crawl :(

(server: dual xeon SQL server!)

[Stadler]

That hack doesn't affect index.php or any other file, except online.php, so this hack is hardly related to this. You have to find out, which requires you to apply the change you've posted above.

btw: which version of MySQL is your Database-Server running?