What I might be able to do is leave blacklisted cache hits, in the cache for 24 or 48 hours, and "no result" hits for 30 minutes. That would help that a bit.

I don't mean to be argumentative, pedigree, but what you seem to be saying is my estimates of central database load and the percentage of queries being blocked by the local cache are wrong somehow. You're the expert. There's no denying that. But I don't see where I went wrong. :confused: With an average of 120 blocked bot registration attempts a day (1,200 bot registrations blocked in 10 days) on my site, that's an average of about 5 bot registration attempts per hour or 2.5 every 30 minutes. If those numbers are right, and based on my log file, they certainly seem to be, how can that 30 minute local cache be blocking very many bots? :erm:

I don't get it. What have I overlooked? Is the problem that bot registration attempts tend to be concentrated (and thus are much higher) at certain times of the day? Is THAT where my logic went wrong?

Thanks!

i just want to say thank you pedigree
its awsome ,i got it running some moths and not ONE spammer came past them dogs
big big thank you ,its a true time saver

You're right, denman. This product is an absolutely great tool.

Searching back through logs = extending cache times but without the headache of parsing another table when I have the mechanism in place already.

Were working an the figures on about 6000 unique IPs generating 600,000 api calls per day. Most arent cached like my mod does. Im only aware of two that are, mine and First defense against spam for wordpress.

At the moment, Im concentrating on finishing the almost total recode of the website, api etc.

Okay. I get it. I think you're saying there's no need to add a log scan when you already have the cache scan working and can easily increase the cache time. Makes sense. Also makes sense to only keep not found items in the cache for 30 minutes or so but keep the found (blacklisted) items in the cache longer. Again, the goal is to reduce the central server load. Why waste its time and energy checking on a bot it rejected an hour or two ago trying to join the same site?

Gotcha. Over time, perhaps you should consider dynamically adjusting the size / duration of the local cache size based on time or the volume of requests processed by that site. That way smaller sites with lower volumes might be able to cache 24 or 48 hours of requests and drastically cut their reliance on the central server while larger / busier sites would not be penailzed because they receive more requests but might rely more heavily on the central server instead. For instance, you might cap the size of local cache at say 300 / 600 entries or 24 / 48 hours whichever came first. It's a thought.

600,000 calls per day is 25,000 calls per hour or 417 calls per minute. With that volume of requests, it's no wonder you're having DNS server outages and server/DB overload problems. I'll bet you cut that load in half by increasing the local server cache time to 12, 24 or 48 hours.

I'll bet Floris and other busy site operators like him would approve of this approach and small site guys (like me) will get better performance and less service outages too.

Good solution! Hope my thinking about this helped. :)

Exact same problem here on v3.8.4 patch level 2.

Have tried browsing to the file from my local machine as well as uploading direct from server.

Why not try the 3.8x version? https://vborg.vbsupport.ru/showthread.php?t=222536

Exact same problem I'm afraid. Sorry, no idea how I ended up in this thread, thanks for the heads-up.

The 3.6 and 3.8 versions are exactly the same and the only difference between them and the 4.0 version is that the 4.0 version has a slightly different mod ID as the forums here dont allow ID duplicates.

Ive never been able to reproduce the invalid file on import as I always import

product-vbstopforumspam.xml

And I have it running myself on 3.8.4 PL2