Weird Problem ... apache? database connection?

[DevilYellow]

This is a thread about what has been going on recently with me

http://www.vbulletin.com/forum/showthread.php?t=207240

Last changes:
11/12/06 installed media wiki
11/11/06 upgraded to vB 3.6.3
11/11/06 upgraded DB Box to FC6 (mysql 5.0.22)
11/11/06 upgraded DB Box RAID firmware
11/11/06 upgraded DB Box mother board bios
11/10/06 mysql restarted (oom-killer)
11/9/06 DB box crashed (RAID / ACCRAID IRQ hard crash)
11/7/06 installed APC Master Switch (remote reboot) restarted both boxes
11/5/06 DB box crashed (RAID / ACCRAID IRQ hard crash)
11/6/06 mysql restarted (oom-killer)
11/3/06 upgraded web box to FC6 (apache 2.2.3/php 5.1.8)
11/1/06 DB box crashed (RAID / ACCRAID IRQ hard crash)
10/28/06 DB box went live

This morning something new happened....three times now it just blanked out and couldnt see the DB. Mysql has been running now for almost 69 hours. Oom-killer has not restarted it. The loads on the web server on average are 0.5 to 1.0 ... starting this morning it bumps up a little to 4 once in awhile. Twice the load has hit 40's and I get a few seconds worth of database error emails. Once I got 50 or so db error messages but the load did not go up. Thru out this the DB box load has not gotten above 0.25.

mysql_connect() [<a href='function.mysql-connect'>function.mysql-connect</a>]: Can't connect to MySQL server on '172.27.0.1' (4) /var/www/html/forums/includes/class_core.php on line 273
MySQL Error :
Error Number :
Date : Monday, November 13th 2006 @ 10:55:18 AM
Script : http://*****.com/forums/
Referrer :
IP Address : 139.62.32.200
Username :
Classname : vB_Database

I am fairly confused as to why this is happening and I am not really thrilled that it happened 3 times in the past 3 hours. It was working perfectly since I left the DC almost 48 hours ago.... the only things I changed is the vBulletin version and added media wiki.

Occassionally the web box goes up to 10 or so for the load .... I have never seen it do that before.... almost like someone is trying to bombard apache every 45 min or so.

[Zachery]

Then try to login to your mysql sever as the vBulletin user.

Its possible there could be an issue on the mysql side.

[DevilYellow]

ERROR 1045 (28000): Access denied for user 'vbuser'@'localhost' (using password: YES)
....it doesnt like my logging in locally....

User Permissions vbuser - hosts 'any'
Datbase Permissions - vbdb - vbuser - any hosts
Host Permissions (was IP but now any host)

I would be confused as to why it works 99.9% of the time otherwise tho.

Went from 11am yesterday on with out having problems ... occasional load spike but never dropped the DB.

I am questioning the apache and mysql configs (http://www.vbulletin.com/forum/showpost.php?p=1251025&postcount=18)...

I did find this in my httpd error_log

[mon Nov 13 08:31:34 2006] [error] [client 24.91.100.190] PHP Fatal error: Allowed memory size of 104857600 bytes exhausted (tried to allocate 71 bytes) in /var/www/html/forums/search.php on line 988, referer: http://www.******.com/forums/showthread.php?t=116710

[mon Nov 13 08:32:28 2006] [error] [client 24.91.100.190] PHP Fatal error: Allowed memory size of 104857600 bytes exhausted (tried to allocate 71 bytes) in /var/www/html/forums/search.php on line 988, referer: http://www.******.com/forums/search.php

[mon Nov 13 08:32:55 2006] [error] [client 24.91.100.190] PHP Fatal error: Allowed memory size of 104857600 bytes exhausted (tried to allocate 71 bytes) in /var/www/html/forums/search.php on line 988, referer: http://www.******.com/forums/search.php

....I put my 2 RSS feeds to 12 hours instead of 30 min and I turned on the Cached Posts Lifespan and set it to 5. Also ran a repair/optimize on all the tables.

I think these things could all help - but I am very curious as to why I was getting such load spikes and couldnt see the DB.

[orban]

Increase the PHP memory limit?

[DevilYellow]

100MB is what its set at right now.... I thought that was plenty.

I really think its something in my Apache Config but not quite sure.

[orban]

Wow 100mb is a lot. I found myself increasing it from 18 to 24mb from php 5.1 to 5.2, but 100! :O

[DevilYellow]

I have a bunch of RAM and the very first box I had was marked as unlimited some how (FreeBSD) ... so I just put an extra 0 behind it and never had a problem in the past 2 years. But this is a new OS and new config setup.

Good news is the DB server has been live with out one single problem for a week.

[DevilYellow]

Update : 12/2

DB box has been running with out any problems for 3 weeks

Web box has been good now for 4 or so days. Yum update and an aggressive httpd.conf seems to have taken away the high loads. PHP memory limit to 50 but I get the occasion fatal exception (white page) where it wants more.

Although it is temporarily stable I am really unsure of it. I have more RAM now to put in the web box and we will see in a few weeks how it does when she is shut down.

[jason|xoxide]

Wow 100mb is a lot. I found myself increasing it from 18 to 24mb from php 5.1 to 5.2, but 100! :O

100MB isn't actually that high, especially for a search. I've got a couple sites that need to be set to 256MB in order for a search to return 1000 hits.

In fact, there's a thread on memory usage right now over at vbulletin.com in the server optimization forum.

[orban]

256mb for a search?! I'll go check out that thread. I ran vB with 12mb, then 18mb and now 24mb (vb 3->3.5->3.6)

[DevilYellow]

<a href="http://www.vbulletin.com/forum/showthread.php?t=210840" target="_blank">http://www.vbulletin.com/forum/showthread.php?t=210840</a>

Seems like something to do with moderator permissions is being called for each thread of the search results. See post 33.

While there are a number of large boards running 3.6 I really hope that my php memory requirements (and to some extent the reason for my load is high) have something to do with this.

Oh, and an update on my situation:
DB Box running 100% strong for 30 days now.
Web Box - no matter what it will overload once in awhile and usually a big over load once every few days. New memory in the box has yet to yield any error.

Have yet to try anything else - but besides RAID configuration and OS/LAMP versions there is nothing else too try.

[orban]

web box..does it swap? and try lighttpd, works so good for me.

[DevilYellow]

Not getting into the swap. I have 4GB in it - going to 6GB soon.

I just dont understand why apache is running this terrible.

How easy is it install, config, and test lighttpd with apache still installed?

Can they use the same directory? I am hoping it would be as easy as install, stop apache, start lighttpd and thats it.

[orban]

Yes I did that. You have to have a PHP with both Apache support + fgci tho before you can switch (unless you run Apache with fgci)

[DevilYellow]

Not sure, how would I check that option for PHP?

[orban]

phpinfo() will tell how it was compiled

[jason|xoxide]

How easy is it install, config, and test lighttpd with apache still installed?

You might want to try Litespeed (http://www.litespeedtech.com/) instead, it's closer to a drop-in replacement for httpd than lighttpd.

[DevilYellow]

litespeed seems pretty cool. I had problems if I had it use the current apache config. It wouldn’t let me change anything - so if I try it again I wont use that option.

I didn’t want to shut down apache to start it just incase there was a problem. I was trying to get it to go on a different port. I may try it again soon. As long as the board would function then it would be great - just need to make sure it logged the same way for webalizer.

What is really starting to piss me off is that apache is still unresponsive 1-4 times a day. Nothing at all in the error_log ... the access_log does not look funny. There is nothing else I can find. I have disabled plugins, tried every combination for the httpd.conf, tried various memory levels for PHP, fixed an error in my hosts file, tried my best to re-create it..... it just doesn’t have any rhyme or reason. Usually once in the morning when people start getting on again - but its not load sensitive. I could have the busiest night and it wouldn’t miss a beat. So why is it having problems at half the load?

I refuse to think its just the fact that its "fedora" - and it scares me that if I do anything with RHEL or CentOS because they are all close I could have the same problem.

With out errors its hard to diagnose :(

Oh, and before upgrading to 3.6.4 the same site was 100% rock solid running 3.5.1 on FC3 with Apache 2.0.52 and PHP 4.3.9

We may have resolution - http://www.vbulletin.com/forum/showpost.php?p=1274028&postcount=28