Some advice relating to hosting

[MrEyes]

Hello all,

I am currently experiencing some problems with my vBulletin server host, and the answers I am getting back from support just don't seem to add up. Not being an expert on these things I need some advice before I start ranting and end up being made to look stupid.

I run a fairly popular forum, average user/guests online is ~120 during normal hours (7am to 1am), dropping to ~5 during early morning hours (1am -> 7am). My current stats are:

• Most users ever online was 170, 26-02-08 at 16:04.
• Threads: 5,162
• Posts: 116,632
• Members: 1,245
• Posts per day: 116,632 / 90 = ~1295
• Threads per day: 5,162 / 90 = ~57

(Division by 90 as the forum is only 3 months old)

I also run the following mods/changes:

• Custom template changes to display Adsense for all logged out users and threads over 15 days old
• Arcade (70 games)
• Casino
• vBulletin Blog
• vBadvanced CMPS

So nothing out of the ordinary there.

Now all this runs on a Linux VPS server with the following relevant setup:

• Disk Space : 15gb (I currently use 24% of this)
• Monthly Bandwidth : 300gb (I currently use ~2.5 gb a day, so I am 75% under the cap)
• Guaranteed Ram : 384mb
• Burstable Ram : 1gb

So that’s the background, on to the actual problem...

Since setting up the server a few months ago, early morning users have reported that they occasionally get the following error:

Fatal error: Out of memory (allocated 786432) (tried to allocate 1966080 bytes) in /home/primarysite/public_html/XXXX on line YYYY

Where XXXX is a random page within the site and YYYY is a random line number. Also the "allocated" and "allocate" numbers are obviously different on each error report.

In the last few days I have started to receive the same report from users who are on the site during normal hours. This tends to happen for 5 to 10 minutes and then the problem disappears for a few days and then comes back.

At face value the cause of this seems obvious, the VPS is running out of memory and moving to a higher specced VPS package will resolve this.

However...

The numbers here just don't add up. Let me explain, this error has been reported (by users) at:

• 5th March 2008 @ ~8pm (GMT) - At this time there were ~100 users/guests online.
• [numerous dates] @ ~1am to ~3am (GMT) - Typically at this time there are no more than 5 users/guests online.

However, the site has been working fine at:

• 26th February 2008 @ 16:04 - At this time there were 170 users/guests online. This is around 60% load more than when the server failed on the 5th, and around 98% more than the early morning outages.
• [most days] @ ~10am -> ~4pm) - Average users/guests online is around 120 (at the time of writing there are 111).

If this was the "face value" memory issue I would not have expected the server to be able to handle the additional load of the 26th February, or the average daily load and would expect it to easy handle the load during early morning hours. However this is not the case which is why I say that the numbers do not add up.

I have to admit that I am no VPS expert, however I am not aware of any cron jobs or processes running at these times, also after reviewing server logs there have been numerous incidents of privvmpages hitting alert levels at the time these reports occur

In addition to this, until setting up the VPS account at the beginning of February 08 this site was running on a no bells and whistles $10 a month shared hosting package and was handling the load without any problems. The reason for moving to VPS was that I have plans to start attracting more users and did not want the community to experience outages. These "plans" are not yet in place so the load currently being placed on the VPS is near as dammit the same load as I was placing on the shared hosting package. I would have expected the VPS to hold up a lot better considering that it is almost 5 times more expensive. I realise that VPS and Shared hosting are two entirely different things, but I am sure you understand where I am coming from when I say that this VPS should be able to handle the same load, and more, than a two bit shared hosting package.

Now all this has been reported to the host and they seem to have ignored the “just doesn’t add up” part and gone down the route of “shared isn’t VPS” (fair enough) and “you will have to upgrade”. They also suggested that the ~5 users at 1am were somehow generating more server load than the 170 that the server coped with only a few days ago. This is a response that I am not entirely happy with, however as mentioned above I need to get my facts straight before going back and ranting at them. I am also more than happy to be told here (as you are independent) that the host is right and I am killing the server with load (although it would be nice to see how these numbers add up).

I realise that one solution would be to find a better host, but I would rather not go there unless absolutely necessary as I don’t want to put the community through another server upheaval after only having done one 5 weeks ago.

So, if you are still with me thanks for reading this far, do you have any suggestions/ideas/comments?

[snakes1100]

Whats your memory limit set at in php.ini?

As i dont know your time zone, typically most server crons run late night at those times you posted, did you check all the cron dirs in etc and inspect each cron file for the times they are running?

[MrEyes]

The resource configs in the php.ini file are:

max_execution_time = 30
max_input_time = 60
memory_limit = 32M

As far as timezone goes I am in GMT and so is the server. Running the "crontab -e" command gives me:

0 1 * * * /scripts/cpbackup
*/15 * * * * /usr/local/cpanel/whostmgr/bin/dnsqueue > /dev/null 2>&1
2,58 * * * * /usr/local/bandmin/bandmin
0 0 * * * /usr/local/bandmin/ipaddrmap
14 0 * * * /usr/local/cpanel/whostmgr/docroot/cgi/cpaddons_report.pl --notify
0 6 * * * /scripts/exim_tidydb > /dev/null 2>&1
*/5 * * * * /usr/local/cpanel/bin/dcpumon >/dev/null 2>&1

checking /etc/cron* gives me:

[root@vps etc]# ls cron*
cron.deny crontab

cron.d:
sysstat

cron.daily:
00-makewhatis.cron logrotate rpm slocate.cron tmpwatch yum.cron

cron.hourly:

cron.monthly:

cron.weekly:
00-makewhatis.cron yum.cron

However I am not sure how to check the exact times these items run at

So there we have it, I don't think there is anything out of the ordinary

[Dismounted]

PHP seems to be running out of memory. This is probably caused by too many scripts running at the same time. I see a backup script being run, maybe this is causing the issue. The temporary fix would be to increase the memory limit.

[snakes1100]

The front part of each line is the time at which these are set to run

0 1 * * * /scripts/cpbackup
*/15 * * * * /usr/local/cpanel/whostmgr/bin/dnsqueue > /dev/null 2>&1
2,58 * * * * /usr/local/bandmin/bandmin
0 0 * * * /usr/local/bandmin/ipaddrmap
14 0 * * * /usr/local/cpanel/whostmgr/docroot/cgi/cpaddons_report.pl --notify
0 6 * * * /scripts/exim_tidydb > /dev/null 2>&1
*/5 * * * * /usr/local/cpanel/bin/dcpumon >/dev/null 2>&1

minute 1-60 hour 0-23 Day of the Month 1-31 the month 1-12 the weekday 0-7

Basically at this point with cpanel as your host panel, you now have a bunch of scripts being ran, if you look at the 1st & 2nd asterisks you will note one is running every 15 minutes, as with your old vhost, they probably had that server set with a larger memory_limit as well. You could tweak the cron times that they run or update the memory_limit

[MrEyes]

Originally Posted by Dismounted

The temporary fix would be to increase the memory limit.

I have to admit that I do not fully understand the problem, however the VPS is capping out the max memory that it is able to use at an OS level, rather than PHP itself running out of allocated memory. So would changing the PHP memory cap actually do anything to resolve this.

To be honest I have gone back to the host and said that, technical issues aside I am paying 5 times more for the VPS than I did for a basic shared hosting package, and everything suggests that the VPS underperforms when compared to the old shared host. Have a look at the attached image, which is from Google Analytics, everything to the left of the red line is from the shared host, everything to the right is the VPS. From this you can see that the cheap as chips shared host was handling more traffic without any issues.

There were two reasons for moving to VPS, primarily I intend to start promoting the site and therefore attract more traffic. On extremely busy days I would occasionally hit the hosts MySQL max concurrent connections limit (which was very low @ 15)

After a full on Google session I have found that everything I currently have on Cron is more or less exactly the same as 90% of other hosts out there. So I am at a loss.

I have a feeling that it might be time to introduce donations/subscriber system to the forum in order to pay for a decent hosting service. The community will love that

[snakes1100]

Have you checked the freemem at those times to see if your account is actually using that much ram that your supposedly allocated in the vps?

Have you checked the shm size for the kernel?

[MrEyes]

Originally Posted by snakes1100

....checked the freemem....

No, because trying to catch this is similar to trying to catch a greased pig. The server will spit out errors at random times under random load on random days, and there is no way I can manually monitor the server 24/7 in the hope that this occurs.

If it was the "face value" memory issue considering that user/guest levels are fairly predictable (between 100 to 140 during normal daytime hours (GMT)) I would expect this to happen fairly regularly, however this just doesnt happen. For example right now there are "129 (100 members and 29 guests)" online and everything is holding up fine.

Originally Posted by snakes1100

Have you checked the shm size for the kernel?

Not heard of that, how do I check this?

[snakes1100]

Well each OS is different, but you can cat /proc/sys/kernel/shmmax, if that spits out an error saying it dont exist, ls /proc/sys/kernel and see what you have in there.

The default should spit back 33554432 = 32mb

You can try echo'ing in a larger value or change it in sysctl.conf

Im guessing that this may be a issue with the overall server of the vps and the ram not being allocated correctly to your vps, is the host US based or no?

[Dismounted]

Originally Posted by MrEyes

I have to admit that I do not fully understand the problem, however the VPS is capping out the max memory that it is able to use at an OS level, rather than PHP itself running out of allocated memory. So would changing the PHP memory cap actually do anything to resolve this.

Fatal error: Out of memory (allocated 786432) (tried to allocate 1966080 bytes) in /home/primarysite/public_html/XXXX on line YYYY

The above error you posted is a PHP error, not an error at the OS level.