File Based Posts Cache : Believe It

Ok, well I sorta have this idea...im not so sure everyone will agree with it.. but in my personal task, I find it to be better.

What I am wanting to do, is keep a flatfile based cache of posts ordered in there own folders via thread id in their own array as they would appear in mysql table.

For example say you want to cache threadid 5454. Now depending on 1 thing, it changes final file output. You determine when you run script how many post per file, before it creates another one.. for example.

Thread 5454 has 5000 post, you want to have 1000 post per file, structure would be:
/(urforum)/cache/5/4/5/4/5454_0.php
/(urforum)/cache/5/4/5/4/5454_1.php
/(urforum)/cache/5/4/5/4/5454_2.php
/(urforum)/cache/5/4/5/4/5454_3.php
/(urforum)/cache/5/4/5/4/5454_4.php

In the case where another thread say 3526 has 500 post, and a limit of 1000 post per file, structure would be:
/(urforum)/cache/3/5/2/6/3526.php

This is all done, to limit the size of each file generated. As most people only have their post per page set anywhere from 20-50 no point in loading one whole thread at once, when you can split it into multiple files.

Now, back to the point, I personally like to grab files directly from the filesystem when possible as to mysqldb, especially if I have some noobs try to DOS me, cause MySQL will always take the hit the worst due to queries. Even so, filebased reads generally will be faster and less cpu intensive.

This is really just an alpha script, to show proof of concept atleast for the part that makes the files..

Basically, you create a 'cache' directory, chmod it 777, upload this file and point ur browser to it like so..

http://ursite.com/urforum/cache/cach...anypostperfile

And vwallah, it creates the files. Hence my own problem thusfar is with memory limit errors, ive tried my best to cut down the memory usage.. however testing on my own server it generally works fine however when i used it on a large thread ie. 12,000 post, it wants to cough up.. Maybe someone here can shed some light.

Also, another interesting sideeffect, if you use the script, set it real low.. like say 50 post per page, on a say 500 post thread, doesnt really matter... an interesting thing happens, if you look at the filesize for each incremented file, you will notice it is getting larger by around 30kb.. even though it should be making a generally steady filesize for each file until the last. As that is the whole point of spreading it amongst files in the first place.

Take a look, tell me what you think etc... Once I figure this out.. I know I will be implementing it, cause I personally like filebased caching. Sure this will probably double the size of ur total forum... but then again if you have a dedicated server like me.. it doesnt really matter..

O by the way, what I had planned on it doing, was everytime a new thread or new post is created, it is auto added to the cache, so no having to manually rebuild the cache everytime.

[Erwin]

This is a great idea and an interesting implementation method. There are lots of potential for uses, and would really reduce server load if it is possible to complete remove the need to connect to the database for normal thread perusing by guests, spiders or users. It would be most useful for close and old threads, which do not need dynamic updates of post information.

I look forward to updates on this concept.

[AWS]

Originally Posted by Erwin

This is a great idea and an interesting implementation method. There are lots of potential for uses, and would really reduce server load if it is possible to complete remove the need to connect to the database for normal thread perusing by guests, spiders or users. It would be most useful for close and old threads, which do not need dynamic updates of post information.

I look forward to updates on this concept.

Well said Erwin my thought exactly. I have said many times that vbulletin needs a true archive which would take closed threads or older threads and put them in flatfiles to rerad from there. This seems to be taking it one step further. To have all threads read from static files would reduce the server load imensly. The script itself after the initial run would be resource intense, but, once it only has to do a few threads say every 15 minutes it wouldn't be bad at all.

I hope this becomes a reality. Someone started something like this for vb2 and never finished it.

so be it... i'm happy my questions are answered well... - i'm a bit devil's advocate so i ask strange things!

... this hack answer my need of an archive way to do things... hope to see the ACP soon!

feature request: archive a complete thread in zip or tar when the topic is closed or deleted... so we can make a followup..

[Xenon]

hmm, i think i know what i'll do in my Holidays ^^

i'll port my vb Archive to vb3

Actually to get back to this hack, i'm with erwin, that seems to be a great idea.

also, once we're in... would be good to have a specific archive browser for this cache... so if we want to keep these cached threads in a different pattern, we can have a different template to display them... i like that idea, not used for everyone though!

[Trigunflame]

Originally Posted by Erwin

This is a great idea and an interesting implementation method. There are lots of potential for uses, and would really reduce server load if it is possible to complete remove the need to connect to the database for normal thread perusing by guests, spiders or users. It would be most useful for close and old threads, which do not need dynamic updates of post information.

I look forward to updates on this concept.

I actually hadnt thought of that erwin, my general idea was for it to be seamlessly integrated into the forum itself, but now that you mention it. Really is a great idea, considering not having to use "any" connection.

Also, if mysql went down the potential would be there for people to still read thread in some cases, if I can put together a script that uses Vbulletins code to reproduce a queryless forum.

Ps. anyone got any ideas on reducing memory usage, or about the increase in file size sequentially. I know its not that big of a deal as we can adjust the sizes for memory limit... but I want this hack to be available to everyone, not just those that can manually tweak their settings.

[Erwin]

Originally Posted by Trigunflame

I actually hadnt thought of that erwin, my general idea was for it to be seamlessly integrated into the forum itself, but now that you mention it. Really is a great idea, considering not having to use "any" connection.

Also, if mysql went down the potential would be there for people to still read thread in some cases, if I can put together a script that uses Vbulletins code to reproduce a queryless forum.

Ps. anyone got any ideas on reducing memory usage, or about the increase in file size sequentially. I know its not that big of a deal as we can adjust the sizes for memory limit... but I want this hack to be available to everyone, not just those that can manually tweak their settings.

Basically, you would be creating a new way of storing threads and posts in parsed html in files, rather than in the mysql database. Very interesting concept! vB3 can already store parsed HTML in the database to save on queries and load, but imagine if you could do this in real-time in files, and really only update the threads daily if a new post is posted... you would cut down server load by a lot, especially for large sites like mine, by eliminating database queries altogether for normal thread reading!

[lazytown]

Some of you are stating that this would reduce server load, but I am confused by this. Programs like UBB Classic originally used a flat-file type system for the entire board. They then upgraded it to "Threads" that uses MYSQL. Threads can handle many more simultaneous users and posts. Itsn't that what also makes VBulletin able to handle many more users & posts? I previously used UBB which did use flat files and it was much more server intensive.

-Victor

PS: Perhaps this is only true if users do searches/etc? I'm almost certain MYSQL will use less resources on a search than flat files.

[Trigunflame]

You are correct vissa in the search aspect, although I have not done realtime benchmarks im almost certain that a fulltext search in mysql would be faster than an ereg or array manipulation THEN search inside the result of a flatfile.

But that is not what this script is doing, you will still be using mysql to search, the post will still be stored in mysql. This script will only make a change to the part of vbulletin that actually shows the post in which case will be drawn from a flatfile WHICH will be faster and less CPU intensive than a query of result rows from MySQL, I should also mention using less Memory.

In your regardst to threads, i doubt that is wh at UBB is referring to. They probably just coined that name in reference to "threads" as in topics. "Real" "Threads" are kernel dependent things. Depends on if your BSD/*Nix/etc build supports "Threading" which extended support can be compiled from within MySQL if you build it yourself.

To support optimized MultiThreading you have to build a kernel with modified File Handles, a new Glibc and latest and greatest LinuxThreads module then build your MySQL using other-libc its a pretty complicated process but lets you optimize the range of MySQL of about 400% or more. In tests ive done as well as various books, its shown that doing a MultiThread modfication can take a fully optimized MySQL from being able to handle several hundred active connections to well over 4000, limited only by your file handle count.

Only problem is, a lot of people dont know that trick and instead rely on getting multiple DB servers, when they can solve their problem with a semi-complex recompile.

Back to the subject, flatfile will always be faster for fetching results, and since these are done in read mode, they can support multiple file reads at once, there is no locking when done with reading.

[Freezerator]

Interesting,

but will discspace grow huge?

For instance: I have almost 900.000 posts and 60.000 threads.
If one cached thread takes up 30kb, it will cost me around 1800mb discspace.

That's almost twice my database size?