vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vBulletin 2.x Full Releases (https://vborg.vbsupport.ru/forumdisplay.php?f=4)
-   -   List Search engine Spider at Who's online (https://vborg.vbsupport.ru/showthread.php?t=54000)

Stadler 06-07-2003 10:00 PM
List Search engine Spider at Who's online
 
I've created this hack, cause I wanted to see, when search engine spiders are browsing my forums.

This hack adds a textbox, where you can enter a list of Spider-IPs, so they would be recognized as 'Search engine spider' on who's online.

I'm using the lists from www.iplists.com right now and it seems to work without problems, though I've had to add some IPs from Inktomisearch that aren't listed there.

First I've used parts of the banip-function, but that was really slow then, so I've had to tweak it. Now its only slightly slower, than without it. You won't even recognize it.

I've ported it to the banip-function in order to reduce server-load, when using a big list of banned IPs. I'll release that, when I'm finished here.

Screenshots:
Search engine spider browsing forums
New textbox to add IPs in Admin-CP

PS: I'd be happy, if you only click install, if you really installed it, since I want to know, how many really installed it and since I need feedback.

I've added a version, that uses the useragent rather, than using an IP-List. I recommend using spiderips_useragent.txt from now on.

Stadler 06-08-2003 01:54 AM
Search engine spider browsing forums

Stadler 06-08-2003 01:55 AM
New textbox to add IPs in Admin-CP

Beanyhead 06-08-2003 02:25 AM
[high]* Beanyhead installs :)[/high]

kaotic 06-08-2003 02:26 AM
Nice hacks. I plan to install both of them tomorrow, as I don't have enough time right now. :(

Stadler 06-08-2003 02:29 AM
That would be good :)

I'm running it for some days on our forums now and have occurred no problems so far, but maybe there are still bugs I haven't occured yet.

Dean C 06-08-2003 09:07 AM
Only problem is this relies on knowing the IP's of the spiders and recognising them ;)

- miSt

Stadler 06-08-2003 01:10 PM
e. g. with the lists from www.iplists.com? ;)

gmarik 06-09-2003 01:44 PM
Great, I got the same idea a week ago ... and than I found it's done by you.

MrNase 06-10-2003 08:00 AM
Hey great hack, clicked install...

but howto? just copy&paste the ip's from iplist.com to acp ?

Dean C 06-10-2003 02:42 PM
Quote:
06-08-03 at 03:10 PM Stadler said this in Post #8
e. g. with the lists from www.iplists.com? ;)
Touch? ;)

monstergamer 06-14-2003 10:26 PM
cool hack
**clicks install**

Freddie Bingham 06-14-2003 10:35 PM
Maybe you want to look at the useragent instead of going by ip (the vb3 way)? It can be spoofed but it is a whole lot simpler and faster. Look at Who's Online on vbulletin.com

preg_match('#(googlebot|lycos|ask jeeves|scooter|fast-webcrawler|slurp@inktomi|turnitinbot)#'

Kriek 07-23-2003 02:00 AM
Test/Install via 2.3.0 failed, no errors, no change on who’s online.

Stadler 07-23-2003 12:42 PM
Unfortunately we're running vB3 now, so I need to install vB2.3.0 locally. Maybe I'll take a look in the next few days, to get it to work with vB2.3.0 and maybe I'll take a look at freddies suggestion, but I suppose I'll hardcode it then, like in vB3 Beta 3 so its easier to install.

Blizzb 07-23-2003 05:51 PM
Does this work for 2.3.0?

Kriek 07-23-2003 06:46 PM
Quote:
Today at 02:51 PM Blizzb said this in Post #16
Does this work for 2.3.0?
*sigh*

Blizzb, read my previous statement. :rolleyes:

Quote:
Yesterday at 11:00 PM Kriek said this in Post #14
Test/Install via 2.3.0 failed, no errors, no change on who?s online.

Till 08-03-2003 11:13 AM
Installation was pretty easy, good job. :)

I added the IPs from iplists.com, where do I see the active users though?

PurpleCow 08-03-2003 12:51 PM
Stadler - Its a nice hack, but can you get it to work on 2.3.0 stable release also ?

Thanks

Stadler 08-03-2003 01:15 PM
Sorry, but I can't find any problems. It worked without problems with a freash installation of vB 2.3.0 for me.

I'll take a look into freddies suggestion, soon. But I can only debug locally.

Till 08-03-2003 09:42 PM
Quote:
Yesterday at 04:15 PM Stadler said this in Post #20
Sorry, but I can't find any problems. It worked without problems with a freash installation of vB 2.3.0 for me.

I'll take a look into freddies suggestion, soon. But I can only debug locally.
Can you tell me where the screenshot is from where it showed the bot spidering in the list of active users?

Stadler 08-03-2003 09:47 PM
Who's Online? ... Click on 'Currently active Users: xx'

in german: 'Zur Zeit aktive Benutzer: xx' ;)

Till 08-03-2003 10:22 PM
Quote:
Today at 12:47 AM Stadler said this in Post #22
Who's Online? ... Click on 'Currently active Users: xx'

in german: 'Zur Zeit aktive Benutzer: xx' ;)
Oh! :D I disabled this one... hehe.

There is a crawler on my forum right now! Yay!!!

Danke. ;)

Megadeuce 09-05-2003 11:45 AM
This hack works well. I wish it could tell me who's spider it is but here is a screenshot....

Doesn't show up on who's online though

UPDATED: must switch on "must resolve IPs" in the admin control panels to see the name of the spider.

Can we get this to also show up on who visited the site today?

Moyses 09-19-2003 05:15 AM
this is a cool hack, doomo arigato

Dominus 10-24-2003 10:25 AM
vbulletin 2.3.2
------------------

Using freddies suggestion, works quite well... ;)

preg_match('#(googlebot|lycos|ask jeeves|scooter|fast-webcrawler|slurp@inktomi|turnitinbot)#'

See screenshot.

Quote:
Originally Posted by Stadler
Unfortunately we're running vB3 now, so I need to install vB2.3.0 locally. Maybe I'll take a look in the next few days, to get it to work with vB2.3.0 and maybe I'll take a look at freddies suggestion, but I suppose I'll hardcode it then, like in vB3 Beta 3 so its easier to install.

Stadler 10-24-2003 08:31 PM
OK, now I've created a version, that works similar to freddies suggestion. But I need someone, who tests it for me. First, that I didn't forgot something in the attached file and second, that it works fine in a live environment.

***The Attachment was moved to the first post***

Rampag33 10-26-2003 05:03 AM
Where's the user-agent box to add ips

I ran the query but don't see it in the admin cp

search the database and don't find anything realted to the query for it.

Dominus 10-26-2003 06:28 AM
Quote:
Originally Posted by Rampag33
Where's the user-agent box to add ips

I ran the query but don't see it in the admin cp

search the database and don't find anything realted to the query for it.
You do not add IPS, you add useragents into this.

PHP Code:
// spider list
$spidernamelist = array(
  'scooter'         => 'Altavista',
  'fast-webcrawler' => 'AlltheWeb',
  'ask jeeves'      => 'Ask Jeeves',
  'googlebot'       => 'Google',
  'grub.org'        => 'Grub Client',
  'slurp@inktomi'   => 'Inktomi',
  'lycos'           => 'Lycos',
  'turnitinbot'     => 'Turnitin.com'
); 

Rampag33 10-26-2003 01:57 PM
OK cool :) damn come on bots

*** installed ***

Dimitrix 03-21-2004 07:59 PM
Great hack,
works like a charm :)

So far only GoogleBots bother me but Ill keep my eyes open :)
Thanks again
Dimitry

Highlander 03-31-2004 03:44 AM
i have a problem with a new searchbot .. ..always when he is coming.. i have the error displayed on the screen and he is messig up the whole db .. i will loose the mysql server if i dont switch the forum offline .. he is making a server load of 7 or more .. normaly when i have 100 users online i have maybe a server load of 0.5 maximum 1.2 ..the sql server crashes when this bot is coming..

what i can do? deinstall this hack?

Highlander 03-31-2004 03:46 AM
in index.php on 219 it writes following code:
PHP Code:
 if (!(preg_match("/$webrobot/i", $loggedinrobots) )) $loggedinrobots .= $webrobot . ", "; 
need help.. quickly please .. it crashes nearly every three ays my server when he comes for a crawl :(

(server: dual xeon SQL server!)

Stadler 03-31-2004 12:17 PM
That hack doesn't affect index.php or any other file, except online.php, so this hack is hardly related to this. You have to find out, which requires you to apply the change you've posted above.

btw: which version of MySQL is your Database-Server running?


All times are GMT. The time now is 03:45 AM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01345 seconds
  • Memory Usage 1,805KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (2)bbcode_php_printable
  • (7)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (34)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 
Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 
Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete