Creating a True Archive

Our big board has now gotten large enough (8+M posts) that we've decided to purge a few million old posts to make the board work reasonably well again. As far as we've been able to tell, the size of the post db has always had a huge effect on the performance on the board, at least at this size.

Two questions:

1.) First, anyone want to suggest why this is a dumb idea, and a better alternative to boost (an already tweaked) performance would be X.

2.) More practically, does anyone have any suggestions for programs or plugins that would let me make some sort of flat-file compilation of the posts we're deleting, or otherwise remove them from the database but keep them available separately?

[briansol]

It's a dumb idea in SEO terms. I wouldn't do it.

What kind of performance problems are you having?

really, 8million records isn't THAT big for mysql.

[ShannonA]

The whole database gets locked due to a slow read of the post db, hundreds of MySQL processes stack up until we hit our limit. Everyone is slowed down, and then people start getting errors afterward.

And whether 8 million records is large in MySQL terms or not, it's clearly large in vBulletin terms, as is readily evidenced by upgrades that have taken more than a day at various times.

[Alfa1]

What kind of box are you on?

[Marco van Herwaarden]

Getting locks does not have to be caused by the size of the table. More likely searches etc.. lock the table. First i would do is to look into a Master/Slave database setup.

[AWS]

Originally Posted by briansol

It's a dumb idea in SEO terms. I wouldn't do it.

What kind of performance problems are you having?

really, 8million records isn't THAT big for mysql.

Care to explain how SEO would be a problem?

I've been experimenting with different ways to archive old posts. The problems I run into are I have to close the forums I don't want to archive, run the program, create links to the archive which is easy if you use the forum link feature, then prune the threads I archived and finally reopen the forums.

I then use another use ASP.net so users can search the static archive. This works, but, is time consuming.

Throwing more hardware at a problem is not the answer. vBulletin should have a true archive feature as well as other enhancements to make it easier for big board owners.

[ShannonA]

Originally Posted by AWS

Throwing more hardware at a problem is not the answer. vBulletin should have a true archive feature as well as other enhancements to make it easier for big board owners.

Agreed. I get very tired of that answer, when vBulletin is choking on database sizes that many other apps would deal with swimmingly.

[cheat-master30]

Originally Posted by AWS

Care to explain how SEO would be a problem?

I've been experimenting with different ways to archive old posts. The problems I run into are I have to close the forums I don't want to archive, run the program, create links to the archive which is easy if you use the forum link feature, then prune the threads I archived and finally reopen the forums.

I then use another use ASP.net so users can search the static archive. This works, but, is time consuming.

Throwing more hardware at a problem is not the answer. vBulletin should have a true archive feature as well as other enhancements to make it easier for big board owners.

Although I don't know that much about SEO, wouldn't the problem be that removing the content from the forums to a read only type archive, if the kind reliant on flat files render any links to said topics elsewhere online broken and lower page rank for those individual pages without using of .htaccess.

[AWS]

If you pruned those threads, which many big board owners have to do anyway, you lose the links anyway. By archiving you create links to the static files. The SE will consider the static pages as new content so it might help with SEO.

Adding a 301 redirect to old content would be easy to accomplish as well without using mod rewrite or .htaccess.

[FlyBoy73]

Yup, the SEO ding really only matters if you delete the content..

There are plenty of Big Boards out there with many more posts than 8 million that seem to be doing fine. If you could monetize that content properly, even if you did throw more hardware at it, I would think it would pay for itself..

That being said, I'd like to see a purely offline static option also.. My reason in a bit different, however... For some of my forums I discontinue but would still like to make that content available without having a license... Seems the only option is to use something like HTTrack to pull it out and then publish it... Though, it is crude, IMO..