Weird Problem ... apache? database connection?

This is a thread about what has been going on recently with me

http://www.vbulletin.com/forum/showthread.php?t=207240

Last changes:
11/12/06 installed media wiki
11/11/06 upgraded to vB 3.6.3
11/11/06 upgraded DB Box to FC6 (mysql 5.0.22)
11/11/06 upgraded DB Box RAID firmware
11/11/06 upgraded DB Box mother board bios
11/10/06 mysql restarted (oom-killer)
11/9/06 DB box crashed (RAID / ACCRAID IRQ hard crash)
11/7/06 installed APC Master Switch (remote reboot) restarted both boxes
11/5/06 DB box crashed (RAID / ACCRAID IRQ hard crash)
11/6/06 mysql restarted (oom-killer)
11/3/06 upgraded web box to FC6 (apache 2.2.3/php 5.1.8)
11/1/06 DB box crashed (RAID / ACCRAID IRQ hard crash)
10/28/06 DB box went live

This morning something new happened....three times now it just blanked out and couldnt see the DB. Mysql has been running now for almost 69 hours. Oom-killer has not restarted it. The loads on the web server on average are 0.5 to 1.0 ... starting this morning it bumps up a little to 4 once in awhile. Twice the load has hit 40's and I get a few seconds worth of database error emails. Once I got 50 or so db error messages but the load did not go up. Thru out this the DB box load has not gotten above 0.25.

mysql_connect() [<a href='function.mysql-connect'>function.mysql-connect</a>]: Can't connect to MySQL server on '172.27.0.1' (4) /var/www/html/forums/includes/class_core.php on line 273
MySQL Error : 
Error Number : 
Date : Monday, November 13th 2006 @ 10:55:18 AM
Script : http://*****.com/forums/
Referrer : 
IP Address : 139.62.32.200
Username : 
Classname : vB_Database

I am fairly confused as to why this is happening and I am not really thrilled that it happened 3 times in the past 3 hours. It was working perfectly since I left the DC almost 48 hours ago.... the only things I changed is the vBulletin version and added media wiki.

Occassionally the web box goes up to 10 or so for the load .... I have never seen it do that before.... almost like someone is trying to bombard apache every 45 min or so.

[Zachery]

Then try to login to your mysql sever as the vBulletin user.

Its possible there could be an issue on the mysql side.

[DevilYellow]

ERROR 1045 (28000): Access denied for user 'vbuser'@'localhost' (using password: YES)
....it doesnt like my logging in locally....

User Permissions vbuser - hosts 'any'
Datbase Permissions - vbdb - vbuser - any hosts
Host Permissions (was IP but now any host)

I would be confused as to why it works 99.9% of the time otherwise tho.

Went from 11am yesterday on with out having problems ... occasional load spike but never dropped the DB.

I am questioning the apache and mysql configs...

I did find this in my httpd error_log

[mon Nov 13 08:31:34 2006] [error] [client 24.91.100.190] PHP Fatal error:  Allowed memory size of 104857600 bytes exhausted (tried to allocate 71 bytes) in /var/www/html/forums/search.php on line 988, referer: http://www.******.com/forums/showthread.php?t=116710

 [mon Nov 13 08:32:28 2006] [error] [client 24.91.100.190] PHP Fatal error:  Allowed memory size of 104857600 bytes exhausted (tried to allocate 71 bytes) in /var/www/html/forums/search.php on line 988, referer: http://www.******.com/forums/search.php

 [mon Nov 13 08:32:55 2006] [error] [client 24.91.100.190] PHP Fatal error:  Allowed memory size of 104857600 bytes exhausted (tried to allocate 71 bytes) in /var/www/html/forums/search.php on line 988, referer: http://www.******.com/forums/search.php

....I put my 2 RSS feeds to 12 hours instead of 30 min and I turned on the Cached Posts Lifespan and set it to 5. Also ran a repair/optimize on all the tables.

I think these things could all help - but I am very curious as to why I was getting such load spikes and couldnt see the DB.

[orban]

Increase the PHP memory limit?

[DevilYellow]

100MB is what its set at right now.... I thought that was plenty.

I really think its something in my Apache Config but not quite sure.

[orban]

Wow 100mb is a lot. I found myself increasing it from 18 to 24mb from php 5.1 to 5.2, but 100! :O

[DevilYellow]

I have a bunch of RAM and the very first box I had was marked as unlimited some how (FreeBSD) ... so I just put an extra 0 behind it and never had a problem in the past 2 years. But this is a new OS and new config setup.

Good news is the DB server has been live with out one single problem for a week.

[DevilYellow]

Update : 12/2

DB box has been running with out any problems for 3 weeks

Web box has been good now for 4 or so days. Yum update and an aggressive httpd.conf seems to have taken away the high loads. PHP memory limit to 50 but I get the occasion fatal exception (white page) where it wants more.

Although it is temporarily stable I am really unsure of it. I have more RAM now to put in the web box and we will see in a few weeks how it does when she is shut down.

[jason|xoxide]

Originally Posted by orban

Wow 100mb is a lot. I found myself increasing it from 18 to 24mb from php 5.1 to 5.2, but 100! :O

100MB isn't actually that high, especially for a search. I've got a couple sites that need to be set to 256MB in order for a search to return 1000 hits.

In fact, there's a thread on memory usage right now over at vbulletin.com in the server optimization forum.

[orban]

256mb for a search?! I'll go check out that thread. I ran vB with 12mb, then 18mb and now 24mb (vb 3->3.5->3.6)