Issues with excessive CPU usage continue...

[ilove2rock]

Hi Everyone,

We recently received some help with the my.cnf file from vB.com and this did seem to help alot, but still we have a lingering issue.

Previously we had to restart the site everyday to keep things running, now its more like every 6 - 10 days. We've not been able to isolate what is causing the excess CPU usage.

I recently installed a proper firewall on the server so I am now receiving email messages which are much more revealing. Here is a typical email that starts 6 - 10 days after a reboot.


User:pnetrock PID:2338 PPID:2177 Run Time:93615(secs) Memory:171232(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:2408 PPID:2020 Run Time:176408(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:2436 PPID:1028 Run Time:136691(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:2491 PPID:2176 Run Time:50384(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:3330 PPID:3278 Run Time:10856(secs) Memory:171232(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:4946 PPID:4815 Run Time:172725(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:4975 PPID:4096 Run Time:133195(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:5426 PPID:5289 Run Time:90067(secs) Memory:171232(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:5659 PPID:5615 Run Time:46853(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:6494 PPID:6396 Run Time:7249(secs) Memory:171232(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:7471 PPID:7104 Run Time:129657(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:7672 PPID:7593 Run Time:43250(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:7741 PPID:6693 Run Time:169171(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:8216 PPID:7930 Run Time:86276(secs) Memory:170816(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:9472 PPID:9235 Run Time:3570(secs) Memory:170976(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:9883 PPID:8086 Run Time:165668(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:10137 PPID:10020 Run Time:126022(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:10882 PPID:10421 Run Time:39665(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:10925 PPID:10763 Run Time:82827(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:12401 PPID:11789 Run Time:122449(secs) Memory:170820(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:13147 PPID:12830 Run Time:62(secs) Memory:171232(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:13489 PPID:13245 Run Time:161994(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:14273 PPID:14227 Run Time:78918(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:14362 PPID:13663 Run Time:35705(secs) Memory:170976(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:14820 PPID:14476 Run Time:118772(secs) Memory:171156(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:16696 PPID:15578 Run Time:158468(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:16945 PPID:16386 Run Time:32246(secs) Memory:171232(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:17112 PPID:16686 Run Time:75653(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:18224 PPID:18101 Run Time:115114(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:18921 PPID:18902 Run Time:28850(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:19398 PPID:19188 Run Time:154836(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:19623 PPID:18914 Run Time:71875(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:21824 PPID:21557 Run Time:68351(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:23251 PPID:21275 Run Time:151167(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:23397 PPID:19880 Run Time:111582(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:24233 PPID:24178 Run Time:64868(secs) Memory:170816(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:24409 PPID:24187 Run Time:25119(secs) Memory:171232(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:25988 PPID:24681 Run Time:107711(secs) Memory:171232(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:26326 PPID:25862 Run Time:147210(secs) Memory:170816(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:27041 PPID:25830 Run Time:21564(secs) Memory:170820(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:27268 PPID:26513 Run Time:61116(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:27937 PPID:27346 Run Time:103986(secs) Memory:171232(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:28650 PPID:28306 Run Time:144025(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:29244 PPID:28796 Run Time:183627(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:29555 PPID:29530 Run Time:18069(secs) Memory:171232(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:29810 PPID:29539 Run Time:57671(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:29958 PPID:29415 Run Time:100669(secs) Memory:170976(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:31842 PPID:31814 Run Time:140444(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:31958 PPID:30620 Run Time:14462(secs) Memory:171232(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:32018 PPID:31495 Run Time:180016(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:32378 PPID:31616 Run Time:54046(secs) Memory:170816(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
User:pnetrock PID:32517 PPID:32393 Run Time:97093(secs) Memory:170976(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php

It seems to me that it is a cron job that is hanging up and causing all the problems.

How can I isolate which cron job? I am willing to pay someone to fix this problem.

Thanks in advance for any advice :)

[RichieBoy67]

Is there no time stamp in the log? Just match it with your cron tasks. Chances are it is something major like a database back up or sitemap gen, something like that.

[kh99]

If you're willing to pay someone to figure it out, you should post a request in the Request for Paid Services forum.

If you're feeling adventurous, you could try editing your includes/functions_cron.php file like this (find this section near the end and replace it with this):
if ($nextitem)
{
if ($nextrun = build_cron_item($nextitem['cronid'], $nextitem))
{
if (($fp = fopen('cronlog.txt', 'a')) !== NULL)
{
fwrite($fp, date(DATE_ATOM)." PID:".getmypid()." start '".$nextitem['filename']."'\n");
fclose($fp);
}
$time_start = microtime(true);
include_once(DIR . '/' . $nextitem['filename']);
$t = microtime(true) - $time_start;
if (($fp = fopen('cronlog.txt', 'a')) !== NULL)
{
fwrite($fp, date(DATE_ATOM)." PID:".getmypid()." end '".$nextitem['filename']."', run time $t\n");
fclose($fp);
}
}
}
else
{
build_cron_next_run();
}



then after a while you hopefully can look at cronlog.txt and find the tasks that aren't ending. Of course you'll still have to figure out why, but at least you'll know where to look.

[Max Taxable]

Go through all your scheduled tasks, turning off those you feel you don't need. See if that helps.

If not and ff you can do without the scheduled tasks completely, turn them off one at a time for a few days until you find the one causing the issue.

[Lynne]

I find it odd there is no timestamp on the errors. If you can get the errors to print with a timestamp, then you can take a look at your Scheduled Tasks and see which one(s) correspond with that timestamp.

[ilove2rock]

ok Thank you all for your replies, I have added the above script so I will post results when I get a few :)

What directory will I find cronlog.txt?

Thanks again, Lisa

[Seven Skins]

To turn them off go to: AdminCP > Scheduled Tasks > Scheduled Task Manager > then from controls on the right (drop down) of the tasks disable them.

[kh99]

What directory will I find cronlog.txt?

It should be in the main forum directory (the one with forum.php).

I made a change to the code after I posted it (I forgot that microtime() needed a parameter to return a number instead of string). So hopefully you copied the code after I made that change, or else the run times will be wrong. It may not matter though, because for the times you posted above it looks like you have things hanging around for days, so you should be able to see that without the run times.

[ilove2rock]

Thank you KH99

I did just paste the info in a couple of hours ago.

I've found the forum.php directory and there is only cron.php. I cant find the log file. Do I have to create an empty document?

[kh99]

Oh, yeah, I'm so used to doing stuff on my test forum that I didn't think of the fact that your web server user might not have permission to create files in that directory. So yeah, you could try creating the empty file.

Another option would be to change the path to somewhere (outside your html directory) where a file can be created. I really should have made the file path a variable, but I didn't so there's two places where it would need to be changed.

It could also be that no scheduled tasks have run yet. For some reason it doesn't log if you use the "Run Now" button the Scheduled Task manager. I guess it uses different code to run the tasks.

[ilove2rock]

ok done... lets see what happens. Ive also restarted the server

[kh99]

See anything in your log file yet? There should be something there, even if it's not showing the problem yet.

[ilove2rock]

no, its still empty. I am not the best with this stuff but I did remember to change the file permissions to read and write. One thing though, the file belongs to root, not home. Is that ok?

[kh99]

Well, I'm not really an admin person, but if the file doesn't belong to the same owner or group as the user the web server runs as, then you'd have to make it writable to everyone. I know on our server (not the one for vbullein.org but the one I'm licensed under) they've got it set up so that any files in the web server's directory are set to read only. So when I want to log stuff to a file like you're doing, I create a directory under my home directory and set it to 0x777 (which is still a little dangerous, I guess, but at least it's not available through the web server).

Anyway, I know it's annoying. I had trouble getting it to work before posting it, and when you don't have any error messages it's hard to know why it's failing. If you decide to abandon this and go a different route I'd understand.

[ilove2rock]

ok, I changed to 7's. Might as well let it run for 24 hours incase its something that is set for midnight or after that is the error....

Thanks again for your help

[kh99]

I figured it was too soon to be sure what's causing the excess cpu time, but I think there are scheduled tasks that run every hour, so I think there should be something in the file after an hour or else there's a problem.

One thought I had: do you have a mod to backup the database via scheduled task? That seems like something that could potentially cause problems.

I also noticed that cron.php calls set_time_limit(0) at the beginning, so those scripts won't time out. But it seems like you could pick some reasonable upper limit and set that instead of having no limit. Of course it won't tell you what the problem is (which would be nice to know because it might be something important), but it would at least limit the cpu usage.

[ilove2rock]

Well its been about 48 hours now and that file is still empty.

I guess we either have something wrong here or I managed to turn off the cron job that was causing the error.

After I had a look through them I did disable 2 or 3. I wont know for sure for about 3 more days if the emails start coming in again.

Also you mentioned the backup script which is something that we installed in root, it is not a vBulletin function.

Thanks KH99 for your advice I will keep you posted

[kh99]

Hmm...unless you turned off all cron jobs then you should have seen things being logged, so I guess there's still some persmission problem or something. If you still want to fool around with it you could try creating a php file with code like this:

<?php
if (($fp = fopen('cronlog.txt', 'a')) !== NULL)
{
fwrite($fp, date(DATE_ATOM)." PID:".getmypid()." start \n");
fclose($fp);
}



Then when you run that from your browser you should see any errors and you can try it until you get the file thing worked out. But it's up to you of course.

One thing I probably should have mentioned, if for some reason there's an error (lke a function used in that code does't exist for some reason) then none of your cron jobs will run, so if you don't want to mess with it any more you should remove that code.

You should be able to go to Statistics & Logs > Schedule Task Log and see if anything has run in the past few days. BTW, the reason I didn't suggest that you just use that log is because I don't think every tasks logs something, and I'm pretty sure most don't log their start and end times.

[ilove2rock]

ok thanks again, I have a house full of company for the next couple of days so I cant focus on this till after christmas.

I did check and scheduled tasks are running in the log.

Thanks again, will post back soon

Happy Holidays everyone ;)