vBulletin Crashing Database each morning

[hilaryl]

Hi Everyone,

I'm going to try and be descriptive as possible, and hopefully get some help with a big issue we are having. We run a fairly big forum, 120,000+ users with over 7 million posts.

We currently are running vBulletin 4.1.2, and it's been running fine. Yesterday morning when trying to access the website, we got a database connection error. This essentially locked us out of accessing anything, when I try to log into phpmyadmin, I get the 'max-user-connections' message. So we had to get our host to restart the sql server.

After doing so, we were able to get back in to phpmyadmin, and could see that the 'searchcore_text' table was marked as crashed and needed to be repaired. The table is 3.6GB in size, so the repair took a very long time.

After repairing, the table no longer had the error on it, but the 'threadviews' table now was saying 'in use'. However there were no processes running on it - we thought it was a false positive.

After coming back in this morning, the website was saying database connection error again and our host is repairing the table again I believe.

Both times, the site seemed to go down around the 4am area.

I'm not sure what our best course of action is, the table is repairing, and saying it's fixed, but then we get database connection error again.

It's almost as if something is running at a specific time in the morning that locks a table, and then there's a massive build of queries trying to access that table, and that causes the connections to become overloaded and everything just shuts down.

What kind of query could be causing this though?

Please let me know what other information you need, we really need to get this sorted.

Thanks,
hilaryl

[Zachery]

First, you should upgrade to 4.2.2
Second, you might want to move to new hardware, constant crashing is a bad sign for a server.
Third, you should invest in vBulletin sphinx from digitalpoint, or another vendor. The sphinx search system is 1000x better than what we're able to offer natively.

[hilaryl]

Hi Zachery,

We are actually on 4.1.9 Patch Level 9 (I was incorrect earlier) - I don't imagine that makes things any better though.

We don't believe it's the hardware, because it's only started happening yesterday - and then again this morning. And there weren't any big changes made the day before the big crash.

I will talk to my host about vBulletin sphinx - but right now, we just need things going again.

We have managed to repair the tables and everything seems to be working now. With the exception of a glitch in certain threads where you can't get to the last page. One example being there are 12 pages, and even if you click on page 12, it only shows you the content of page 11. It even says page 12 in the URL.

I've read that this can fixed by Rebuilding Thread Information. Though there are 420,000 threads - should this be done with the forum offline?

Are there any other steps I should take after repairing database tables to kind of refresh the forum? Anything to do with the System Cache perhaps?

Thanks,
hilaryl

[Zachery]

You should really upgrade to 4.2.2.

[ForceHSS]

If you have all security updates there is really no need to update unless the version you have is causing the crashes. Check server error logs and post them here

[ozzy47]

There is security fixes in 4.2.2 pl2 that is not in earlier versions, that is why it is advised to upgrade.

[hilaryl]

Hi ForceHSS,

We do have the latest security updates - which is why we don't believe it's the version causing the issue.

In regards to the server error logs, I'll have a look through and see if there is anything worth posting.

My question before amount maintenance after a table repair, is there any advice anyone has for that?

Thanks,
hilaryl

[AndrewSimm]

Is it crashing around the time you have a scheduled backup?

[ForceHSS]

Originally Posted by hilaryl

Hi ForceHSS,
In regards to the server error logs, I'll have a look through and see if there is anything worth posting.

Thanks,
hilaryl

If anything is in the server error logs even if you think its not worth posting post it as it might help

[hilaryl]

Originally Posted by AndrewSimm

Is it crashing around the time you have a scheduled backup?

Actually yes, the back-up happens around the same time.

My host has moved the scheduled backup to an earlier time for tonight - seeing if that makes a difference.

Do you know of an issue that relates to the backup?