vb.org Archive - 3.8.7 Patch Level 3 Custom Cron Task

vb.org Archive (https://vborg.vbsupport.ru/index.php)

- vB3 General Discussions (https://vborg.vbsupport.ru/forumdisplay.php?f=111)

- - 3.8.7 Patch Level 3 Custom Cron Task (https://vborg.vbsupport.ru/showthread.php?t=302648)

3.8.7 Patch Level 3 Custom Cron Task

Hey all,

I am working on a product that involves querying the database for, say, 100 posts and does stuff with it. I set a scheduled task via Admin Panel to run this task every minute. When the task runs, it causes our forum (front-end) to be unusable until the query is done. I can see how this can be a problem, but why do the built-in vBulletin's cron tasks do not have this problem? Are these cron tasks built using special cron functions? If so, can you point me to the right direction so I can modify my script?

Thank you in advance. I'm looking forward to your responses.

- S

From the admin popup help page:

Quote:

Set it to * to have it run every minute (not really recommended!).

Also, to know why it may be doing that, we would need the code you are using in order to debug it.

Maybe your query is looping but as Ozzy47 says we need to see your code in order to help with that.

Thanks for your responses. I was able to figure it out. The code was going through too many loops.

Thanks, all!

Thanks for replying back, and glad you got it sorted. :)

Sorry that I have to revive this thread. I figured it would be better to than making a new one. We host a Q/A board and want to spell correct all the posts in our database. There are approximately 3.2 million. Our cron task (query for 1000 posts + sent to our spell correct engine) takes about 30 seconds to complete one task. The problem is that when it kicks off, it renders the page unusable so the user ends up having to wait until the task is complete. There are tons of cron jobs that vBulletin also runs, but it doesn't seem to hang the page. What are some reasons why the page would become unusable? I thought the cron job was suppose to run in the background. If not, how can I make it "run in the background" like a real cron?

Thank you in advance.

-S

As we said before, post your code so we can see what you are doing.

Just a bit of background info: the goal of this function is correct the text that gets sent to it. The function utilizes regex patterns to determine what needs fixing and when.

I can't post the full code, but I will try to post enough so that it gives you guys enough information to have an idea of what could be going on. Do keep in mind that it takes about 30 seconds to complete correcting 1000 posts. The 30 secs may be normal, but since it's running as a cron job, it should not render the page unusable since it's running in the background...at least it should be. Just in case you guys are wondering where's the query, I've tested that the query completes about 1/10 of a second or even faster. The "stall" occurs when the page text gets sent to the fix_spelling function. Anyways, here's the code:

Code:

<?php



public function fix_spelling($text, $severity = 0, $exceptions = array()) {



    list($rgx_start, $rgx_end_replace) = 

        array('(?<=[ .,!?>\]\r\n]|^)', '(?=[ .,!?<\[\r\n]|$)');



    $rgx_end_delete = '( |(?=[.,!?\[\r\n]|$))';



    // $this->dictionary is a multi-dimensional array filled with ~4000 elements.

    // An element consists of multiple attributes, (corrected_word, error_weight and others)



    foreach($this->dictionary as $word=>$prop) {



            // this line might be problematic due to in_array being called 4000 times

            if(stripos($text, $word) !== FALSE && !in_array($word, $exceptions)) {



                    // some initialization based on value of $prop[corrected_word] takes

                    // place here ....finally, we call a preg-replace on a match



                    $text = preg_replace("variables that contain regex goes here (i.e. $rgx_start and so on)", "corrected_word goes here", $text, -1, $count);



                    if($count) {

                            // We want to highlight the corrected words so users know what was corrected

                            // Initialize highlight with another preg_replace goes here...syntax is similar to $text

                            $highlight = preg_replace("....");

                    }

            }

    }



    // we want to capitalize i but there are some cases where we shouldn't, such as i.e.

    // do a preg_match for some special cases

    $rgx_list = preg_match("....");

    // call a preg_replace again on $text

    $text = preg_replace("....$rgx_list, ....");



    // if replacements occurred $count is set

    if($count)

            $highlight = preg_replace(".....");



    // now we also want to have special corrections in case the post is a Math-based post

    if(preg_match("regex patterns for math keywords such as sin, pi, theta and so on")) {

            // do something ..jsut a variable initialization

    }



    // fix single letters such as b, n, r, u, y => be, and, are, you, why respectively

    list($rgx1, $rgx_start, $rgx_end) = array('regex pattern1', 'regex pattern2', 'regex pattern3');



    // this foreach loop should only run 5 times...the length of the array holding the "b, n, r, u, y"

    foreach ($letters as $letter=>$fix) {

            // There are two if statements which does two preg_replace each...the 2nd preg_replace will only happen if the 1st preg_replace found replacements (i.e. $count is set)



    }



    ....

    // There are a total of 8 more preg_replace calls total and 2 preg_matches

    .....

    return array(array($text, $highlight), $error_score);

}

It would be prudent for you to check your server logs and see if you are using a lot of memory or cpu time when this runs, it doesn't matter that it's "running in the background" you still need the resources for the task to take place.

Thank you for the response. I'll keep that in mind while I continue to troubleshoot.

-S

Have you considered running this as a true cron job and not a vBulletin Scheduled Task?

And, you said it does 1000 posts... is this every single time... every minute? Why 1000? If it's an arbitrary number, have you considered writing to the database the last postid you did so that you only need to run this for new posts instead of 1000 posts each time?

Lynne,

Running it as a true cron job would be ideal since it can just run for hours. I was under the impression that I couldn't do that with vBulletin. I know how to set up true cron jobs, but how would I set it up such that it will run through vBulletin? Any documentation can help. I wasn't able to find anything that related to my problem via search.

Regarding your question, it actually does grab 1000 new posts each time. We have a flag called "fixed" and that flag is set to 1 for posts that have been corrected. We skip those. The reason why we're doing this is because we want to correct all posts in the database. A lot of posts were posted way back since ~2005 or possibly earlier (I just came on board a few months ago). Our current engine corrects new posts, but we want to correct all the previous posts (~3.2 million) of them. Choosing 1000 would make sense since we want to be done with it at a reasonable time.

I'm looking forward to your help with setting this up as true cron job that will work via vBulletin.

Thank you in advance.

You wouldn't need to run it "through" vbulletin, your cron is working on the database which is a stand alone that vbulletin queries, so unless your spell engine is only integrated in to vbulletin and cant be used as a stand alone you should be fine to set up a true cron.

You may want to see how this person did it - https://vborg.vbsupport.ru/showthread.php?t=164477 There really isn't a reason you can't run it as a real cron job. My guess would be you just need to add some lines of code to connect to the database and then the rest of your code should work fine.

I'd suggest doing smaller posts in batches, if your post table is myisam, you might want to change it to innodb, while this task is running anyway.

Thank you everyone for your responses. I'll consider these options and will report back on the results.

--------------- Added [DATE]1383870166[/DATE] at [TIME]1383870166[/TIME] ---------------

Simon,

The spell correct engine is a plugin for vBulletin hence why I need it to run via vBulletin.

--------------- Added [DATE]1383871817[/DATE] at [TIME]1383871817[/TIME] ---------------

Quote:

Originally Posted by Lynne (Post 2459004)

This plugin actually works great, but the only downfall is that it will not work on Windows platform. My boss wants it to work on all systems so unfortunately this solution will not work for my case...although it's a clever plugin. Thanks, Lynne.

--------------- Added [DATE]1383872819[/DATE] at [TIME]1383872819[/TIME] ---------------

From more troubleshooting, it appears the issue is related to an existing plugin. I'll have to figure out which one.

Thank you everyone for your help.

X vBulletin 3.8.12 by vBS Debug Information
Template Usage: (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (1)bbcode_code_printable (2)bbcode_quote_printable (1)footer (1)gobutton (1)header (1)headinclude (6)option (1)post_thanks_navbar_search (1)printthread (16)printthreadbit (1)spacer_close (1)spacer_open Phrase Groups Available: global postbit showthread	Included Files: ./printthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/class_bbcode_alt.php ./includes/class_bbcode.php ./includes/functions_bigthree.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete printthread_start bbcode_fetch_tags bbcode_create bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete printthread_post printthread_complete
Messages:

Phrase Groups Available:

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.01128 seconds Memory Usage 1,766KB Queries Executed 10 (?)
More Information
Template Usage: (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (1)bbcode_code_printable (2)bbcode_quote_printable (1)footer (1)gobutton (1)header (1)headinclude (6)option (1)post_thanks_navbar_search (1)printthread (16)printthreadbit (1)spacer_close (1)spacer_open Phrase Groups Available: global postbit showthread	Included Files: ./printthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/class_bbcode_alt.php ./includes/class_bbcode.php ./includes/functions_bigthree.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete printthread_start bbcode_fetch_tags bbcode_create bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete printthread_post printthread_complete
Messages: