File Based Posts Cache : Believe It

Ok, well I sorta have this idea...im not so sure everyone will agree with it.. but in my personal task, I find it to be better.

What I am wanting to do, is keep a flatfile based cache of posts ordered in there own folders via thread id in their own array as they would appear in mysql table.

For example say you want to cache threadid 5454. Now depending on 1 thing, it changes final file output. You determine when you run script how many post per file, before it creates another one.. for example.

Thread 5454 has 5000 post, you want to have 1000 post per file, structure would be:
/(urforum)/cache/5/4/5/4/5454_0.php
/(urforum)/cache/5/4/5/4/5454_1.php
/(urforum)/cache/5/4/5/4/5454_2.php
/(urforum)/cache/5/4/5/4/5454_3.php
/(urforum)/cache/5/4/5/4/5454_4.php

In the case where another thread say 3526 has 500 post, and a limit of 1000 post per file, structure would be:
/(urforum)/cache/3/5/2/6/3526.php

This is all done, to limit the size of each file generated. As most people only have their post per page set anywhere from 20-50 no point in loading one whole thread at once, when you can split it into multiple files.

Now, back to the point, I personally like to grab files directly from the filesystem when possible as to mysqldb, especially if I have some noobs try to DOS me, cause MySQL will always take the hit the worst due to queries. Even so, filebased reads generally will be faster and less cpu intensive.

This is really just an alpha script, to show proof of concept atleast for the part that makes the files..

Basically, you create a 'cache' directory, chmod it 777, upload this file and point ur browser to it like so..

http://ursite.com/urforum/cache/cach...anypostperfile

And vwallah, it creates the files. Hence my own problem thusfar is with memory limit errors, ive tried my best to cut down the memory usage.. however testing on my own server it generally works fine however when i used it on a large thread ie. 12,000 post, it wants to cough up.. Maybe someone here can shed some light.

Also, another interesting sideeffect, if you use the script, set it real low.. like say 50 post per page, on a say 500 post thread, doesnt really matter... an interesting thing happens, if you look at the filesize for each incremented file, you will notice it is getting larger by around 30kb.. even though it should be making a generally steady filesize for each file until the last. As that is the whole point of spreading it amongst files in the first place.

Take a look, tell me what you think etc... Once I figure this out.. I know I will be implementing it, cause I personally like filebased caching. Sure this will probably double the size of ur total forum... but then again if you have a dedicated server like me.. it doesnt really matter..

O by the way, what I had planned on it doing, was everytime a new thread or new post is created, it is auto added to the cache, so no having to manually rebuild the cache everytime.

before trying to help or test, i have some questions:

1- serverside, what is the cost in resources ?!... because reading files is more resource-expensive than read the SQL cache...

2- moving thread, make a real mess in your cache files ?!

3- archive browsing ?!

4- on Windows server, that load of directories would crash the server... solution ?!

5- what is the real use of this anyway ?!.... real points, not only dreaming that it may work!

my personal need for such a system would be to archive - really archive - closed threads... so when a thread is locked or closed, instead of browsing it from the DB, it would read the files in a text way, like most blogs do...

so ursite.com/thread/1/53 would load the post #53 in the thread #1 ...

[Trigunflame]

1. actually no.. mysql is a hell of a lot more resource intensive than reading a file into an array from a flatfile

2. why would that make a mess, the post data is the only thing stored..and thats done by thread id... doesnt matter what forum u move the thread to oO

3. i dont see why not

4. its using the same concept vbulletin does for folder limits.... by seperating them into 4/3/5/6/7/ etc...

5. to save resources.... a direct filebased read will always be quicker in most cases especially under load, then having to send a query to mysql, and mysql fetch data, then send it back through link

So yes..there is a valid reason..

ps. dont double post.

[Andreas]

Hmm .. how would you enforce permissions on viewing threads with this?
Just everybody could access the files ...

And it would be kinda static - no display of ppl currently browsing the thread, no online indicators, ...

[Trigunflame]

Kirby...what are you talking about, cache.php is just a proof of concept file to generate the cache.

The post data are stored in serialized arrays of all the data that is normally present in the rows of the post table, line for line.

The actual code I will make will be implemented directly into the showthread/newpost functions to handle reading from the cache instead of from the mysqldb.

Its not like im trying to remake a whole forum, Please understand, all this does is save Post array data to a flatfile, this does not mess with permissions or any of that crap, its just simply using the files to read the data, instead of from the mysql db.

Note: The post will still be stored in the db, it just wont be read from there when people view threads..

[Andreas]

Originally Posted by Trigunflame

Basically, you create a 'cache' directory, chmod it 777, upload this file and point ur browser to it like so

Originally Posted by Trigunflame

Please understand, all this does is saved Post data to a flatfile, this does not mess with permissions or any of that crap, its just simply using the files to read the data, instead of from the mysql db.

AFAIK 777 is world-readable, so just anyone could view the thread - no matter if he has permission to do so or not.

[Trigunflame]

Lol, Kirby... its not that hard to write a <?php die; ?> at the top of the files, to keep anything from being shown through the browser. Or did you not know that trick? :ermm:

ps. 777 is "word-writable/readable"

[Andreas]

Ack. That's the point I missed in your description
I though you are just generating data-files (although I should have noticed the .php - lol).

Anyway, I'd really fear the idea of heaving some 100.000 thread-files floating around.

> ps. 777 is "word-writable/readable"
Hmm ... how does Word come into play here?

[Trigunflame]

Originally Posted by KirbyDE

Ack. That's the point I missed in your description
I though you are just generating data-files (although I should have noticed the .php - lol).

Anyway, I'd really fear the idea of heaving some 100.000 thread-files floating around.

> ps. 777 is "word-writable/readable"
Hmm ... how does Word come into play here?

When you make typos from staying up nearly 2 days straight like me OO