How to set a different session timeout for spiders?

[AceNZ]

I would like to set a different session timeout for spiders than for logged-in users. Spiders don't usually accept cookies, so it can quickly look like you have hundreds of them simultaneously accessing the site, when in fact it's just a single spider making a lot of requests. Reducing the session timeout will help minimize that effect.

I suspect that spiders aren't identified until the who is online page is displayed -- so it would actually be OK to make this change for all guests.... Ideally, the code would detect whether an incoming request included a session cookie, and set the timeout based on that.

Does anyone know where I would look in the code to make such a change?

[Lynne]

Why not just set your crawl-delay in your robots.txt file?

[AceNZ]

Originally Posted by Lynne

Why not just set your crawl-delay in your robots.txt file?

I don't want to reduce the speed at which the site is crawled. I want to reduce (or ideally coalesce) the redundant / meaningless guest users in the who is online list.

--------------- Added [DATE]1196639979[/DATE] at [TIME]1196639979[/TIME] ---------------

I think I've figured out half of the answer (and I'm starting to understand vB's plugin system at the same time). I added the following code to the who's online online_query hook:

PHP Code:

if($vbulletin->options['WOLguests']){
  $rkdatecut = TIMENOW - 300;
  $hook_query_where = " and session.userid <> 0 or (session.userid = 0 and session.lastactivity > $rkdatecut)";
} 


The seems to have the effect I'm looking for in online.php. Now I'm looking for a similar hook for the summary that's displayed on the forum home page.

--------------- Added [DATE]1196640977[/DATE] at [TIME]1196640977[/TIME] ---------------

Unfortunately, there doesn't seem to be a hook in the associated query in the forumhome code, so I had to modifiy the code directly instead. Not ideal, but it works.

PHP Code:

$datecut = TIMENOW - $vbulletin->options['cookietimeout'];
$rkcut = TIMENOW - 300;
$numbervisible = 0;
$numberregistered = 0;
$numberguest = 0;

$forumusers = $db->query_read_slave("
        SELECT
            user.username, (user.options & " . $vbulletin->bf_misc_useroptions['invisible'] . ") AS invisible, user.usergroupid,
            session.userid, session.inforum, session.lastactivity,
            IF(displaygroupid=0, user.usergroupid, displaygroupid) AS displaygroupid, infractiongroupid
        FROM " . TABLE_PREFIX . "session AS session
        LEFT JOIN " . TABLE_PREFIX . "user AS user ON(user.userid = session.userid)
        WHERE (session.lastactivity > $datecut and session.userid <> 0) or (session.lastactivity > $rkcut and session.userid = 0)
        " . iif($vbulletin->options['displayloggedin'] == 1 OR $vbulletin->options['displayloggedin'] == 3, "ORDER BY username ASC") . "
    ");