Issues with excessive CPU usage continue...

[ilove2rock]

Hi Everyone,

We recently received some help with the my.cnf file from vB.com and this did seem to help alot, but still we have a lingering issue.

Previously we had to restart the site everyday to keep things running, now its more like every 6 - 10 days. We've not been able to isolate what is causing the excess CPU usage.

I recently installed a proper firewall on the server so I am now receiving email messages which are much more revealing. Here is a typical email that starts 6 - 10 days after a reboot.

Usernetrock PID:2338 PPID:2177 Run Time:93615(secs) Memory:171232(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:2408 PPID:2020 Run Time:176408(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:2436 PPID:1028 Run Time:136691(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:2491 PPID:2176 Run Time:50384(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:3330 PPID:3278 Run Time:10856(secs) Memory:171232(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:4946 PPID:4815 Run Time:172725(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:4975 PPID:4096 Run Time:133195(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:5426 PPID:5289 Run Time:90067(secs) Memory:171232(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:5659 PPID:5615 Run Time:46853(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:6494 PPID:6396 Run Time:7249(secs) Memory:171232(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:7471 PPID:7104 Run Time:129657(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:7672 PPID:7593 Run Time:43250(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:7741 PPID:6693 Run Time:169171(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:8216 PPID:7930 Run Time:86276(secs) Memory:170816(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:9472 PPID:9235 Run Time:3570(secs) Memory:170976(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:9883 PPID:8086 Run Time:165668(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:10137 PPID:10020 Run Time:126022(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:10882 PPID:10421 Run Time:39665(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:10925 PPID:10763 Run Time:82827(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:12401 PPID:11789 Run Time:122449(secs) Memory:170820(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:13147 PPID:12830 Run Time:62(secs) Memory:171232(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:13489 PPID:13245 Run Time:161994(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:14273 PPID:14227 Run Time:78918(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:14362 PPID:13663 Run Time:35705(secs) Memory:170976(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:14820 PPID:14476 Run Time:118772(secs) Memory:171156(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:16696 PPID:15578 Run Time:158468(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:16945 PPID:16386 Run Time:32246(secs) Memory:171232(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:17112 PPID:16686 Run Time:75653(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:18224 PPID:18101 Run Time:115114(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:18921 PPID:18902 Run Time:28850(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:19398 PPID:19188 Run Time:154836(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:19623 PPID:18914 Run Time:71875(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:21824 PPID:21557 Run Time:68351(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:23251 PPID:21275 Run Time:151167(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:23397 PPID:19880 Run Time:111582(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:24233 PPID:24178 Run Time:64868(secs) Memory:170816(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:24409 PPID:24187 Run Time:25119(secs) Memory:171232(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:25988 PPID:24681 Run Time:107711(secs) Memory:171232(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:26326 PPID:25862 Run Time:147210(secs) Memory:170816(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:27041 PPID:25830 Run Time:21564(secs) Memory:170820(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:27268 PPID:26513 Run Time:61116(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:27937 PPID:27346 Run Time:103986(secs) Memory:171232(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:28650 PPID:28306 Run Time:144025(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:29244 PPID:28796 Run Time:183627(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:29555 PPID:29530 Run Time:18069(secs) Memory:171232(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:29810 PPID:29539 Run Time:57671(secs) Memory:171076(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:29958 PPID:29415 Run Time:100669(secs) Memory:170976(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:31842 PPID:31814 Run Time:140444(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:31958 PPID:30620 Run Time:14462(secs) Memory:171232(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:32018 PPID:31495 Run Time:180016(secs) Memory:171072(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:32378 PPID:31616 Run Time:54046(secs) Memory:170816(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php
Usernetrock PID:32517 PPID:32393 Run Time:97093(secs) Memory:170976(kb) exe:/usr/bin/php cmd:/usr/bin/php /home/pnetrock/public_html/cron.php

It seems to me that it is a cron job that is hanging up and causing all the problems.

How can I isolate which cron job? I am willing to pay someone to fix this problem.

Thanks in advance for any advice

[RichieBoy67]

Is there no time stamp in the log? Just match it with your cron tasks. Chances are it is something major like a database back up or sitemap gen, something like that.

[kh99]

If you're willing to pay someone to figure it out, you should post a request in the Request for Paid Services forum.

If you're feeling adventurous, you could try editing your includes/functions_cron.php file like this (find this section near the end and replace it with this):

PHP Code:

    if ($nextitem)
    {
        if ($nextrun = build_cron_item($nextitem['cronid'], $nextitem))
        {
            if (($fp = fopen('cronlog.txt', 'a')) !== NULL)
            {
               fwrite($fp, date(DATE_ATOM)." PID:".getmypid()." start '".$nextitem['filename']."'\n");
               fclose($fp);
            }
            $time_start = microtime(true);
            include_once(DIR . '/' . $nextitem['filename']);
            $t = microtime(true) - $time_start;
            if (($fp = fopen('cronlog.txt', 'a')) !== NULL)
            {
               fwrite($fp, date(DATE_ATOM)." PID:".getmypid()."   end '".$nextitem['filename']."', run time $t\n");
               fclose($fp);
            }
        }
    }
    else
    {
        build_cron_next_run();
    } 


then after a while you hopefully can look at cronlog.txt and find the tasks that aren't ending. Of course you'll still have to figure out why, but at least you'll know where to look.

[Max Taxable]

Go through all your scheduled tasks, turning off those you feel you don't need. See if that helps.

If not and ff you can do without the scheduled tasks completely, turn them off one at a time for a few days until you find the one causing the issue.

[Lynne]

I find it odd there is no timestamp on the errors. If you can get the errors to print with a timestamp, then you can take a look at your Scheduled Tasks and see which one(s) correspond with that timestamp.

[ilove2rock]

ok Thank you all for your replies, I have added the above script so I will post results when I get a few

What directory will I find cronlog.txt?

Thanks again, Lisa

[Seven Skins]

To turn them off go to: AdminCP > Scheduled Tasks > Scheduled Task Manager > then from controls on the right (drop down) of the tasks disable them.

[kh99]

Originally Posted by ilove2rock

What directory will I find cronlog.txt?

It should be in the main forum directory (the one with forum.php).

I made a change to the code after I posted it (I forgot that microtime() needed a parameter to return a number instead of string). So hopefully you copied the code after I made that change, or else the run times will be wrong. It may not matter though, because for the times you posted above it looks like you have things hanging around for days, so you should be able to see that without the run times.

[ilove2rock]

Thank you KH99

I did just paste the info in a couple of hours ago.

I've found the forum.php directory and there is only cron.php. I cant find the log file. Do I have to create an empty document?

[kh99]

Oh, yeah, I'm so used to doing stuff on my test forum that I didn't think of the fact that your web server user might not have permission to create files in that directory. So yeah, you could try creating the empty file.

Another option would be to change the path to somewhere (outside your html directory) where a file can be created. I really should have made the file path a variable, but I didn't so there's two places where it would need to be changed.

It could also be that no scheduled tasks have run yet. For some reason it doesn't log if you use the "Run Now" button the Scheduled Task manager. I guess it uses different code to run the tasks.