Some people have very restrictive cookie settings (keep until the browser closes) or regularly wipe them manually, but you're probably right in that this is a small percentage.
Any chance you could separate recognised spiders from other IPs where the log cut-off date is concerned? The reason I've started using your mod is for antispam purposes, to see if any unrecognised crawlers are poking around our forums and I bet yahoo's slurp spider alone will end up producing half the log size. Would help if you could only save the last 24 hours of known spider IPs (for the display) and log everything elsefor a longer period.
|