File Based Posts Cache : Believe It

Ok, well I sorta have this idea...im not so sure everyone will agree with it.. but in my personal task, I find it to be better.

What I am wanting to do, is keep a flatfile based cache of posts ordered in there own folders via thread id in their own array as they would appear in mysql table.

For example say you want to cache threadid 5454. Now depending on 1 thing, it changes final file output. You determine when you run script how many post per file, before it creates another one.. for example.

Thread 5454 has 5000 post, you want to have 1000 post per file, structure would be:
/(urforum)/cache/5/4/5/4/5454_0.php
/(urforum)/cache/5/4/5/4/5454_1.php
/(urforum)/cache/5/4/5/4/5454_2.php
/(urforum)/cache/5/4/5/4/5454_3.php
/(urforum)/cache/5/4/5/4/5454_4.php

In the case where another thread say 3526 has 500 post, and a limit of 1000 post per file, structure would be:
/(urforum)/cache/3/5/2/6/3526.php

This is all done, to limit the size of each file generated. As most people only have their post per page set anywhere from 20-50 no point in loading one whole thread at once, when you can split it into multiple files.

Now, back to the point, I personally like to grab files directly from the filesystem when possible as to mysqldb, especially if I have some noobs try to DOS me, cause MySQL will always take the hit the worst due to queries. Even so, filebased reads generally will be faster and less cpu intensive.

This is really just an alpha script, to show proof of concept atleast for the part that makes the files..

Basically, you create a 'cache' directory, chmod it 777, upload this file and point ur browser to it like so..

http://ursite.com/urforum/cache/cach...anypostperfile

And vwallah, it creates the files. Hence my own problem thusfar is with memory limit errors, ive tried my best to cut down the memory usage.. however testing on my own server it generally works fine however when i used it on a large thread ie. 12,000 post, it wants to cough up.. Maybe someone here can shed some light.

Also, another interesting sideeffect, if you use the script, set it real low.. like say 50 post per page, on a say 500 post thread, doesnt really matter... an interesting thing happens, if you look at the filesize for each incremented file, you will notice it is getting larger by around 30kb.. even though it should be making a generally steady filesize for each file until the last. As that is the whole point of spreading it amongst files in the first place.

Take a look, tell me what you think etc... Once I figure this out.. I know I will be implementing it, cause I personally like filebased caching. Sure this will probably double the size of ur total forum... but then again if you have a dedicated server like me.. it doesnt really matter..

O by the way, what I had planned on it doing, was everytime a new thread or new post is created, it is auto added to the cache, so no having to manually rebuild the cache everytime.

that's what i said... used for archiving, not pure editing/search.

what i would suggest is to focus on one part of the possibilities... some here are suggesting it can become a great archive system, for closed topics and archive system, and i think it can be the best of both worlds... because actually it's one of the things that is missing most here...

also, would be cool to merge with a semi-RSS feed, like on most blogs ... once we control this technique, personal blogs for our members will be easy to deal with, as there is no need to use the db for blogs...

anyway, back to work/test!

[Erwin]

Originally Posted by vissa

Some of you are stating that this would reduce server load, but I am confused by this. Programs like UBB Classic originally used a flat-file type system for the entire board. They then upgraded it to "Threads" that uses MYSQL. Threads can handle many more simultaneous users and posts. Itsn't that what also makes VBulletin able to handle many more users & posts? I previously used UBB which did use flat files and it was much more server intensive.

-Victor

PS: Perhaps this is only true if users do searches/etc? I'm almost certain MYSQL will use less resources on a search than flat files.

UBB classic used CGI to serve up dynamic information based on a flat-file system. It's the CGI/ PERL-based code that is the drain on the server.

However, what we are talking here is static HTML files, not CGI/ PHP based - that is a world of difference to UBB. We are talking about loading up static HTML files.

Great for threads to be read by guests who can't post anyway.

Mind you, I'm sure forum programmers would have thought of this, and if it was this simple to implement, it would have been done already. I'm sure there would be some overhead, which will probably come from the frequent updating of these static HTML files, which would probably override any if not all of the savings in server resources.

[Zachery]

Originally Posted by Erwin

UBB classic used CGI to serve up dynamic information based on a flat-file system. It's the CGI/ PERL-based code that is the drain on the server.

However, what we are talking here is static HTML files, not CGI/ PHP based - that is a world of difference to UBB. We are talking about loading up static HTML files.

Great for threads to be read by guests who can't post anyway.

Mind you, I'm sure forum programmers would have thought of this, and if it was this simple to implement, it would have been done already. I'm sure there would be some overhead, which will probably come from the frequent updating of these static HTML files, which would probably override any if not all of the savings in server resources.

I could see issues with the dynamic content for posts, if there was a constant writing to the file system it would be almost as bad as using mysql if not worse

[TosaInu]

Interesting.

Is it possible to use more than one SQL database from another server within same network? We have a large Of Topic forum and it would be nice if the topics and posts of that one were served from the other database/server we have.

[buro9]

Nice thread, I'm a bit of a caching geek so let me point you to a few things:

Firstly... take a look at the cache cannon hack I was making for VB 2.x, it was abandoned as I was making it for a very specific person who was on a safe_mode protected box and it wouldn't work with safe_mode... however it did work fine on a normal box.

Anyhow, link for that:
https://vborg.vbsupport.ru/showthread.php?t=36000

That created 100% flat filed and browsable forums, threads and posts.

For searchability you would have to consider something that indexed the threads seperately (such as a site indexing tool) or simply embed a Google search box

Secondly, if you want to improve caching for existing items but leave them in the database, consider using one of the PEAR cache libraries:
http://pear.php.net/packages.php?cat...atname=Caching

Thirdly... and possibly the most interesting... dump your posts, threads and forums data to XML files, embed at the top of these the path of an applicable XSLT file to render them, and offer an XPATH way to search, sort and browse them.

That is technically possible and feasible, though it will depend on your skills with XML, XSLT and XPATH as to whether you can make that work. I tell you one thing though... it would be a hell of a thing to see and would allow you to change the presentation of it over time in a way that static HTML would not.

The XSLT transformation gets shifted onto the browser btw... server side transformation would be intensive to say the least.

Helpful starters there:
A SQL 2 XML package:
http://pear.php.net/package/XML_sql2xml

XML Transformer package:
http://pear.php.net/package/XML_Transformer

XML XPath Queries package:
http://pear.php.net/package/XML_XPath