archiving to the filesystem!

[krohnathlonman]

I believe I own several of the larger postcount forums and instead of pruning threads like I was doing which resulted in traffic loss from search engines I am coding a way to archive old threads into the filesystem to cleanup the DB! this actually even gets opened up through the /archive/ files but not through showthread.php yet.

I'm wondering how many people are interested in this.... Cause I can release a beta pretty soon

[Andreas]

Hmm .. how large is your forum?
Mine is approx. 1.6 million posts and the DB isn't thaat big.

[krohnathlonman]

largest is about 13million posts..... the one I'm provessing now is about 4.5M and had a nearly 7Gb DB... I've already trimmed 3.5Gb and its still rolling....

This mod is ONLY for the super big I guess I have the archive set by date so that you can archive based on date. I'm finding 200 days back is a good way to handle it. I'm unsure of load and things like that... I won't know till next week.

[Marco van Herwaarden]

File system posts will have more impact on performance then posts stored in the database. Also is it more difficult to tune this.

I don't think storing in the file system would be the way to go for this.

[krohnathlonman]

Have you ever tried to make a backup of a 10Gb+ DB? Then how about making alterations to the post table? The upgrade to VB 3.5 was an absolute nightmare with the mods to the post table.

The only time there can be any possible impact on performance is if some of the archives get MAJOR traffic. What I'm doing is only archiving threads that are over 1yr old and in some case older than 200 days. What this does is makes the DB alot easier to manage while still having the contents in a nice and easy format for the search engines. If you know of something better for the truly HUGE boards I'd like to hear about it because I've been trying many things the past few months and archiving to the filesystem is by far the best option.

[Marco van Herwaarden]

Still it would make more sense to arvhive them to a second database (on another server?) then to store them in the filesystem.

Also solutions like load balanced servers and things like would be more the way to approach this i guess. Create a load balanced situation with replication servers, and run the backups from the replica (or consider the replica as the backup, use advanced operating system backup solutions first to a filer and from there to offline media).