List Search engine Spider at Who's online

I've created this hack, cause I wanted to see, when search engine spiders are browsing my forums.

This hack adds a textbox, where you can enter a list of Spider-IPs, so they would be recognized as 'Search engine spider' on who's online.

I'm using the lists from www.iplists.com right now and it seems to work without problems, though I've had to add some IPs from Inktomisearch that aren't listed there.

First I've used parts of the banip-function, but that was really slow then, so I've had to tweak it. Now its only slightly slower, than without it. You won't even recognize it.

I've ported it to the banip-function in order to reduce server-load, when using a big list of banned IPs. I'll release that, when I'm finished here.

Screenshots:
Search engine spider browsing forums
New textbox to add IPs in Admin-CP

PS: I'd be happy, if you only click install, if you really installed it, since I want to know, how many really installed it and since I need feedback.

I've added a version, that uses the useragent rather, than using an IP-List. I recommend using spiderips_useragent.txt from now on.

[Till]

Yesterday at 04:15 PM Stadler said this in Post #20
Sorry, but I can't find any problems. It worked without problems with a freash installation of vB 2.3.0 for me.

I'll take a look into freddies suggestion, soon. But I can only debug locally.

Can you tell me where the screenshot is from where it showed the bot spidering in the list of active users?

[Stadler]

Who's Online? ... Click on 'Currently active Users: xx'

in german: 'Zur Zeit aktive Benutzer: xx'

[Till]

Today at 12:47 AM Stadler said this in Post #22
Who's Online? ... Click on 'Currently active Users: xx'

in german: 'Zur Zeit aktive Benutzer: xx'

Oh! I disabled this one... hehe.

There is a crawler on my forum right now! Yay!!!

Danke.

[Megadeuce]

This hack works well. I wish it could tell me who's spider it is but here is a screenshot....

Doesn't show up on who's online though

UPDATED: must switch on "must resolve IPs" in the admin control panels to see the name of the spider.

Can we get this to also show up on who visited the site today?

[Moyses]

this is a cool hack, doomo arigato

[Dominus]

vbulletin 2.3.2
------------------

Using freddies suggestion, works quite well...

preg_match('#(googlebot|lycos|ask jeeves|scooter|fast-webcrawler|slurp@inktomi|turnitinbot)#'

See screenshot.

Originally Posted by Stadler

Unfortunately we're running vB3 now, so I need to install vB2.3.0 locally. Maybe I'll take a look in the next few days, to get it to work with vB2.3.0 and maybe I'll take a look at freddies suggestion, but I suppose I'll hardcode it then, like in vB3 Beta 3 so its easier to install.

[Stadler]

OK, now I've created a version, that works similar to freddies suggestion. But I need someone, who tests it for me. First, that I didn't forgot something in the attached file and second, that it works fine in a live environment.

***The Attachment was moved to the first post***

[Rampag33]

Where's the user-agent box to add ips

I ran the query but don't see it in the admin cp

search the database and don't find anything realted to the query for it.

[Dominus]

Originally Posted by Rampag33

Where's the user-agent box to add ips

I ran the query but don't see it in the admin cp

search the database and don't find anything realted to the query for it.

You do not add IPS, you add useragents into this.

PHP Code:

// spider list
$spidernamelist = array(
  'scooter'         => 'Altavista',
  'fast-webcrawler' => 'AlltheWeb',
  'ask jeeves'      => 'Ask Jeeves',
  'googlebot'       => 'Google',
  'grub.org'        => 'Grub Client',
  'slurp@inktomi'   => 'Inktomi',
  'lycos'           => 'Lycos',
  'turnitinbot'     => 'Turnitin.com'
); 


[Rampag33]

OK cool damn come on bots

*** installed ***