[Megareus Rex]

Originally Posted by mikelbeck

Yeah, that's way too much. I'll have to take another look at that.

I know how to fix it, the problem is I don't want to do it that way. What happens is this... When a guest arrives at a page, the spider plug in writes a line to the database. Just the timestamp, user agent and page. When you run the spiders page, it "rolls up" all of the data so it's displayed in a nice, neat format and all the old data is removed. That's what's creating so many queries.

If I were to change the plug in to check to see if there's already a record for that spider in the database and then just update it as the page is loaded, there would be no need to "roll up". But I think that would add a few queries on each page load (by a non-user, meaning a guest or spider) and I don't think that's a good idea.

I'll see what else I can come up with.

Just letting you know, its almost at 1100 queries, and a page generation time of 7+ seconds.

Not to mention the Yahoo Slurp spider's # of IPs has more than doubled (perhaps close to tripled) from earlier.