How to set a different session timeout for spiders?

[AceNZ]

I would like to set a different session timeout for spiders than for logged-in users. Spiders don't usually accept cookies, so it can quickly look like you have hundreds of them simultaneously accessing the site, when in fact it's just a single spider making a lot of requests. Reducing the session timeout will help minimize that effect.

I suspect that spiders aren't identified until the who is online page is displayed -- so it would actually be OK to make this change for all guests.... Ideally, the code would detect whether an incoming request included a session cookie, and set the timeout based on that.

Does anyone know where I would look in the code to make such a change?

[Lynne]

Why not just set your crawl-delay in your robots.txt file?

[AceNZ]

Why not just set your crawl-delay in your robots.txt file?

I don't want to reduce the speed at which the site is crawled. I want to reduce (or ideally coalesce) the redundant / meaningless guest users in the who is online list.

--------------- Added 1196639979 at 1196639979 ---------------

I think I've figured out half of the answer (and I'm starting to understand vB's plugin system at the same time). I added the following code to the who's online online_query hook:

if($vbulletin->options['WOLguests']){
$rkdatecut = TIMENOW - 300;
$hook_query_where = " and session.userid <> 0 or (session.userid = 0 and session.lastactivity > $rkdatecut)";
}

The seems to have the effect I'm looking for in online.php. Now I'm looking for a similar hook for the summary that's displayed on the forum home page.

--------------- Added 1196640977 at 1196640977 ---------------

Unfortunately, there doesn't seem to be a hook in the associated query in the forumhome code, so I had to modifiy the code directly instead. Not ideal, but it works.

$datecut = TIMENOW - $vbulletin->options['cookietimeout'];
$rkcut = TIMENOW - 300;
$numbervisible = 0;
$numberregistered = 0;
$numberguest = 0;

$forumusers = $db->query_read_slave("
SELECT
user.username, (user.options & " . $vbulletin->bf_misc_useroptions['invisible'] . ") AS invisible, user.usergroupid,
session.userid, session.inforum, session.lastactivity,
IF(displaygroupid=0, user.usergroupid, displaygroupid) AS displaygroupid, infractiongroupid
FROM " . TABLE_PREFIX . "session AS session
LEFT JOIN " . TABLE_PREFIX . "user AS user ON(user.userid = session.userid)
WHERE (session.lastactivity > $datecut and session.userid <> 0) or (session.lastactivity > $rkcut and session.userid = 0)
" . iif($vbulletin->options['displayloggedin'] == 1 OR $vbulletin->options['displayloggedin'] == 3, "ORDER BY username ASC") . "
");