Why not do 1 feed then post at a time rather than save in memory?

[John Diver]

Hey,

I wanted my RSS poster to go through each feed one by one, I.e. get the feed, post then go to the next.

I think right now it is all being saved in memory for all feeds at once then posted once complete.

It is causing problems with my hosting timing out or simply too much in the memory, meaning all feeds that have been retrieved are lost and it only picks up the feeds after the last run date.

Is there any other way to run the feeds so it doesnt use so much memory?

Thanks

[kh99]

You would need to change includes/cron/rssposter.php. It looks like it could be restructured to do one feed at a time and free the memory in between, but if the problem is time then that won't help. Another possibly easier thing to do might be to make a number of copies of rssposter.php, modify the query at the beginning so that it each copy only gets one or more feeds (by filtering on the ids), then set up each file to run as a separate scheduled task at but at different times.

[John Diver]

Good thinking Kevin on doing separate files :)

It would work easier in the long run to edit the rssposter but I think I would need someone to code that, PHP isn't my thing, I can only fix small errors unfortunately.

Thanks Kevin :)

[kh99]

I'm not sure if you're asking, but I think it would just be something like this (existing code starts around line 34, I added the part in red):

// ################################################## ###########################
// slurp all enabled feeds from the database

$feeds_result = $vbulletin->db->query_read("
SELECT rssfeed.*, rssfeed.options AS rssoptions, user.*, forum.forumid
FROM " . TABLE_PREFIX . "rssfeed AS rssfeed
INNER JOIN " . TABLE_PREFIX . "user AS user ON (user.userid = rssfeed.userid)
INNER JOIN " . TABLE_PREFIX . "forum AS forum ON (forum.forumid = rssfeed.forumid)
WHERE rssfeed.options & " . $vbulletin->bf_misc_feedoptions['enabled'] . "
AND rssfeed.rssfeedid IN (1, 2, 3)
");



where of course you'd change the 1, 2, 3 for each file. You'd also want to disable the original rssposter.php task.

I don't have time to set things up to test it, but if you want to try it, I think it should work.

[John Diver]

I wasn't asking you to do it, but thank you so much for helping Kevin :)

I am going to test this today and I will reply here with the results.

Thanks once again Kevin, really appreciate you helping me :)

--------------- Added 1353074786 at 1353074786 ---------------

Quick question, what would happen if a feed was removed in the admincp but it was still included in the rssposter file?

For example I was checking and I have a few feeds that haven't been updated in a long time (This is for feeds on a personal site) and I removed them in admincp, but haven't removed the reference in the rssposter file - Would it stop or crash anything?

I would really like if vB updated the Rssposter - It is the only thing I have been having major problems with for a few years now as I always use it.
I have been told by my host that I have to upgrade to a VPS just because of the RSS poster because it is using too much memory when running..I can't afford it for a personal site that is just for my own blog feeds..

I was hoping this would be resolved in vb4 and was happy to upgrade just for this, but it also does the same :(

[kh99]

If you remove a feed nothing will happen, because the query just says "get every row that matches these feed ids", so if one id doesn't match any row there will just be one less row (feed) to process, and if none match it shouldn't do anything. But it's true that it's a little difficult to manage. I was going for something that would be easy to implement.

Maybe something else could be done to make it a little better, like maybe modifying rssposter to run only the N 'oldest' feeds (in terms of the last run time), then you wouldn't need to worry about feed ids.

As for vB updating the rss poster - I haven't looked to see what it looks like in vb5 (if it's there yet at all), but I believe development on vb4 is finished except for security fixes (I think there's a vb4.2.1 with some minor fixes that will be released some time, but after that it's mostly done).

[kh99]

Actually what I mentioned above is a better idea and is easier to implement (I guess that's what happens when you think about it for a minute before replying :o )

Anyway, I think this should work to limit the number of feeds that run each time:

// ################################################## ###########################
// slurp all enabled feeds from the database

$feeds_result = $vbulletin->db->query_read("
SELECT rssfeed.*, rssfeed.options AS rssoptions, user.*, forum.forumid
FROM " . TABLE_PREFIX . "rssfeed AS rssfeed
INNER JOIN " . TABLE_PREFIX . "user AS user ON (user.userid = rssfeed.userid)
INNER JOIN " . TABLE_PREFIX . "forum AS forum ON (forum.forumid = rssfeed.forumid)
WHERE rssfeed.options & " . $vbulletin->bf_misc_feedoptions['enabled'] . "
ORDER BY lastrun ASC LIMIT 1
");


Now you don't have to make extra copies of rssposter.php, just modify the original. You would want to change it to run much more often, and if you have many feeds you may need to change the "LIMIT 1" to something higher. I guess this method has the disadvantage that you have to make sure that the task runs often enough to handle all your feeds.

[John Diver]

Thanks again Kevin!

Sorry, but I am trying to think of how this will work, for example, say I have 50 feeds (I have been adding a few over the years :) ) - Would

ORDER BY lastrun ASC LIMIT 1 run each of the 50 if I set it up as a cron job? (I have got the cron job working now by calling cron.php and just the RSSposter task)

Would this work if 1 of my feeds has 50 posts that haven't been retrieved, post those, then go on to the next feed?

Sorry Kevin, like I said, not very good with PHP, just trying to understand how it would work :)

Thank you once again!

--------------- Added 1353079405 at 1353079405 ---------------

I forgot to say, the reason for calling the poster by cron was because the site is just for me, so it doesn't get users that would trigger the cronimage to run the cronjob to post feeds

[kh99]

Would this work if 1 of my feeds has 50 posts that haven't been retrieved, post those, then go on to the next feed?

What I think it *should* do (without having tested it or knowing exactly what issues you're having), is process one feed every time the task runs (I suppose it will post whatever new items there are, based on the settings in the feed manager). This means if you have 50 feeds the task would need to run 50 times before getting them all. So you could either set the task to run once per minute (and set you cron job as well) and get them over the course of a hour, or increase the LIMIT to do more feeds per run. Of course this will also be affected by the "Check Feed Every..." setting in the feed manager - a feed will only be processed if enough time has passed.

I realize this still isn't the perfect solution you're looking for, but again it's something you can do with a simple edit.

[John Diver]

This is great Kevin, you have really helped :) Really

I am going to do this on a test forum now to see how it goes.

Once again, many thanks for helping me out Kevin :)

[John Diver]

Hey Kevin,

Just getting a chance to do this now, been so busy.

Should I just keep my cron to run rssposter and do it for each minute and keep the limit to 1?
What would happen if the previous feed is taking more than 1 minute to retrieve, should it cause an error?

Thanks again Kevin!

--------------- Added 1353368318 at 1353368318 ---------------

Sorry Kevin, I just noticed I have this in the vb4 forum!

I am using vb3 - This was the only thing I wanted to upgrade to vb4 but it didn't change.

I am not sure, but I think this is the code from vb3 that has been changed for vb4:

// ################################################## ###########################
if ($_POST['do'] == 'updatestatus')
{
$vbulletin->input->clean_gpc('p', 'enabled', TYPE_ARRAY_UINT);

$feeds_result = $db->query_read("SELECT * FROM " . TABLE_PREFIX . "rssfeed ORDER BY title");
while ($feed = $db->fetch_array($feeds_result))
{
$old = ($feed['options'] & $vbulletin->bf_misc_feedoptions['enabled'] ? 1 : 0);
$new = ($vbulletin->GPC['enabled']["$feed[rssfeedid]"] ? 1 : 0);

if ($old != $new)
{
$feeddata =& datamanager_init('RSSFeed', $vbulletin, ERRTYPE_ARRAY);
$feeddata->set_existing($feed);
$feeddata->set_bitfield('options', 'enabled', $new);
$feeddata->save();
}
}

exec_header_redirect('rssposter.php');
}

print_cp_header($vbphrase['rss_feed_manager']);

I will try to get it working through this and post it in the vb3 forum if it will help any other users.

[kh99]

Should I just keep my cron to run rssposter and do it for each minute and keep the limit to 1?

I think that should work.


What would happen if the previous feed is taking more than 1 minute to retrieve, should it cause an error?

I'm not sure, but I think it should work. But if you have one feed taking longer than 1 minute it might time out because there's usually a time limit set for how long a php script can run. If that happens I suppose you could limit the number of items for that feed. If you have one feed that continually has so many items that it takes more than 1 minute to run every time, then I'm not sure what you could do, you might need to figure out a way to rewrite rssposter.php.


I did a diff of the vb3 and vb4 rssposter.php and there are only minor differences, so I wouldn't worry about that issue.

[John Diver]

Thanks Kevin :)

I don't think it should take more than 1 minute to run the feeds, I am going to try it now.

Should the code be changed like this:

// ################################################## ###########################
if ($_POST['do'] == 'updatestatus')
{
$vbulletin->input->clean_gpc('p', 'enabled', TYPE_ARRAY_UINT);

$feeds_result = $db->query_read("SELECT * FROM " . TABLE_PREFIX . "rssfeed ORDER BY title");
while ($feed = $db->fetch_array($feeds_result))
{
$old = ($feed['options'] & $vbulletin->bf_misc_feedoptions['enabled'] ? 1 : 0);
$new = ($vbulletin->GPC['enabled']["$feed[rssfeedid]"] ? 1 : 0);

if ($old != $new)
{
$feeddata =& datamanager_init('RSSFeed', $vbulletin, ERRTYPE_ARRAY);
$feeddata->set_existing($feed);
$feeddata->set_bitfield('options', 'enabled', $new);
ORDER BY lastrun ASC LIMIT 1
$feeddata->save();
}
}

exec_header_redirect('rssposter.php');
}

print_cp_header($vbphrase['rss_feed_manager']);

As you can see, I am obviously useless at PHP ;)

Thanks Kevin for all the help

[kh99]

That's the wrong file: you want to change includes/cron/rssposter.php. It should look just like what I posted above.

[John Diver]

Opps...Red face...

Thanks Kevin.

Just doing this now

--------------- Added 1353448949 at 1353448949 ---------------

Hey Kevin,

Been working with this all day - The code you posted works :)

I changed the limit to 10 because my feeds weren't updating enough - It worked for a while but then suddenly stopped.

Even by running the RSS feeds manually, nothing updated - When I unchecked all feeds, then checked a few and run the update manually, they worked.

Not sure why it would be doing this..

When I run manually with the code included at a limit of 10 it never gets an error or times out, it just shows the page after checking feeds but with no feeds retrieved, I'm not sure why it would be happening.

[John Diver]

Hey,

Just wanted to know if anyone can help please, still having the same problem :(