Consistent VB4 downtime over 5 days - Where do I go from here?

[Fergal C]

I'm extremely frustrated by what's happening on our business forum, that I've put a huge amount of time and money into. It runs on VB4 and is on an expensive VPS with Servint. Over the last five days the forum has been losing data from the database and hence going down regularly. Unfortunately this is very damaging to the community and our search engine rankings. The current downtime is the worst and it has now been down all day - a minimum of over eight hours since I first looked at it this morning.

I've opened a support ticket with VB.com and they have been very helpful. They feel that the issue is happening because of issues with the hosting account. The issue has been escalated to Servint's Engineers who have also been helpful, but they say its a vBulletin issue and that everything is running as it should on the hosting account.

I've paid a reputable and knowledgeable VB Admin to look at the problem and as of yet he hasn't been able to figure out the cause. He has a mirror of the site running on his machine and it's running fine, plus we have a test site on the same server which is also running fine.

These issues might be a little less frustrating if I use using free software on a shared hosting account, but I have not been shy about putting the needed resources into the forum and it has had severe downtime over the last five days.

Any ideas as to what I need to do to determine the cause of these issues?

How can I ensure that something like this doesn't happen again?

Thanks!

[Lynne]

When you say you are "losing data from the database", do you mean that literally - one minute it is there and the next it is not? Have you and your host looked into the possibility that you are being hacked? That someone is just playing with you?

[Fergal C]

Thanks Lynne, yes the host has looked at that possibility and they say that there is no evidence of the site being hacked.

[realmr]

is it possible u have a cron job running for ur particular domain alone ? Have u checked disabling add-ons and enabling them back on after a day or 2 to find the difference ? There could be an outdated add-on which could be causing the issue.

[Digital Jedi]

Did you ever consider going back to a copy of the database from before the problems started occurring? True, that might be a last resort depending on the situation. But if you haven't had good up time since the problems started, maybe the loss will be negligible. At the least, you narrow down the potential list of problems.

[Fergal C]

...Have u checked disabling add-ons and enabling them back on after a day or 2 to find the difference ? There could be an outdated add-on which could be causing the issue.
That's something we are doing now, thanks for the suggestion.

Did you ever consider going back to a copy of the database from before the problems started occurring? True, that might be a last resort depending on the situation. But if you haven't had good up time since the problems started, maybe the loss will be negligible. At the least, you narrow down the potential list of problems.
Thanks for the suggestion. We have been running now since yesterday evening without any major problems. However, that is something we will consider seriously if we run into problems again.

It now appears that the issue may have been caused by some old data that was backed up on the VPS. I am currently working with Servint to have this data deleted and hopefully that will resolve the issue once and for all.

[Digital Jedi]

That's something we are doing now, thanks for the suggestion.


Thanks for the suggestion. We have been running now since yesterday evening without any major problems. However, that is something we will consider seriously if we run into problems again.

It now appears that the issue may have been caused by some old data that was backed up on the VPS. I am currently working with Servint to have this data deleted and hopefully that will resolve the issue once and for all.
Yeah, sometimes it's better to shut down temporarily then try to run with errors, particularly if they're compounding over time. Much harder for monetized sites to have to do, but it can save you and your users some frustration in the long run. Hope everything gets worked out.

[Fergal C]

Today is the ninth day of serious downtime on my forum. My host Servint persisted in blaming the problems on the vBulletin software and only today said;

One of the causes may be the size of your /tmp directory. If the tmp directory fill up while the mysql socket is still trying to write information, then the database can become corrupt. I've removed the size of /tmp by remounting it on your server. I'll need to restore the databases from backup to get them working again

I've had a VPS with Servint for the last seven years and I have to say that I'm extremely disappointed by their lack of support over the last nine days. Servint are not a cheap host and downtime like that is not good enough for a forum that I have invested so much in.

[SocialNetworkBuzz]

Did you run repair on your database? Did you try restarting your VPS? Sometimes we over look the simplest things, resolved by a database repair or machine reboot =)

[SCRIPT3R]

Why does your homepage address redirect to http://www.businessadviceforum.com/install/install.php? Might want to look into that. ;)

[Fergal C]

Thanks Stabatha and ScripT3R we did all of those things and they haven't been working. Servint seem to have identified what's causing the problem and hopefully they will fix it soon.

[SCRIPT3R]

Thanks Stabatha and ScripT3R we did all of those things and they haven't been working. Servint seem to have identified what's causing the problem and hopefully they will fix it soon.
No you didn't, it's still redirecting to your intallation.php... in the wrong hands, this will cause all the errors you are describing. I can access your installation files right now and probably do some damage if I wanted to. This means anyone else can too.

[borbole]

Indeed, I loaded the page and it is redirecting to the installer but it shows a firefox error message instead of the installation page. But I agree, the installer should not be left like that where everyone can access it as it poses a security risk.

[Fergal C]

The error may have appeared to have been caused by a re-direct, but it was actually caused by a DB issue. The site is now back up and running and Servint have confirmed that;

...the size of the /tmp directory was the issue. By default on our servers /tmp is it's own secure partition that is 100Mb in size. [Engineer's name removed] disabled the secure partition. This means that /tmp was bundled into the main server partition and so can use as much space as is left on your server. This should stop this from occurring again.