Why not do 1 feed then post at a time rather than save in memory?

[John Diver]

Hey,

I wanted my RSS poster to go through each feed one by one, I.e. get the feed, post then go to the next.

I think right now it is all being saved in memory for all feeds at once then posted once complete.

It is causing problems with my hosting timing out or simply too much in the memory, meaning all feeds that have been retrieved are lost and it only picks up the feeds after the last run date.

Is there any other way to run the feeds so it doesnt use so much memory?

Thanks

[kh99]

You would need to change includes/cron/rssposter.php. It looks like it could be restructured to do one feed at a time and free the memory in between, but if the problem is time then that won't help. Another possibly easier thing to do might be to make a number of copies of rssposter.php, modify the query at the beginning so that it each copy only gets one or more feeds (by filtering on the ids), then set up each file to run as a separate scheduled task at but at different times.

[John Diver]

Good thinking Kevin on doing separate files

It would work easier in the long run to edit the rssposter but I think I would need someone to code that, PHP isn't my thing, I can only fix small errors unfortunately.

Thanks Kevin

[kh99]

I'm not sure if you're asking, but I think it would just be something like this (existing code starts around line 34, I added the part in red):

// #############################################################################
// slurp all enabled feeds from the database

$feeds_result = $vbulletin->db->query_read("
	SELECT rssfeed.*, rssfeed.options AS rssoptions, user.*, forum.forumid
	FROM " . TABLE_PREFIX . "rssfeed AS rssfeed
	INNER JOIN " . TABLE_PREFIX . "user AS user ON (user.userid = rssfeed.userid)
	INNER JOIN " . TABLE_PREFIX . "forum AS forum ON (forum.forumid = rssfeed.forumid)
	WHERE rssfeed.options & " . $vbulletin->bf_misc_feedoptions['enabled'] . "
           AND rssfeed.rssfeedid IN (1, 2, 3)
");

where of course you'd change the 1, 2, 3 for each file. You'd also want to disable the original rssposter.php task.

I don't have time to set things up to test it, but if you want to try it, I think it should work.

[John Diver]

I wasn't asking you to do it, but thank you so much for helping Kevin

I am going to test this today and I will reply here with the results.

Thanks once again Kevin, really appreciate you helping me

--------------- Added [DATE]1353074786[/DATE] at [TIME]1353074786[/TIME] ---------------

Quick question, what would happen if a feed was removed in the admincp but it was still included in the rssposter file?

For example I was checking and I have a few feeds that haven't been updated in a long time (This is for feeds on a personal site) and I removed them in admincp, but haven't removed the reference in the rssposter file - Would it stop or crash anything?

I would really like if vB updated the Rssposter - It is the only thing I have been having major problems with for a few years now as I always use it.
I have been told by my host that I have to upgrade to a VPS just because of the RSS poster because it is using too much memory when running..I can't afford it for a personal site that is just for my own blog feeds..

I was hoping this would be resolved in vb4 and was happy to upgrade just for this, but it also does the same

[kh99]

If you remove a feed nothing will happen, because the query just says "get every row that matches these feed ids", so if one id doesn't match any row there will just be one less row (feed) to process, and if none match it shouldn't do anything. But it's true that it's a little difficult to manage. I was going for something that would be easy to implement.

Maybe something else could be done to make it a little better, like maybe modifying rssposter to run only the N 'oldest' feeds (in terms of the last run time), then you wouldn't need to worry about feed ids.

As for vB updating the rss poster - I haven't looked to see what it looks like in vb5 (if it's there yet at all), but I believe development on vb4 is finished except for security fixes (I think there's a vb4.2.1 with some minor fixes that will be released some time, but after that it's mostly done).

[kh99]

Actually what I mentioned above is a better idea and is easier to implement (I guess that's what happens when you think about it for a minute before replying )

Anyway, I think this should work to limit the number of feeds that run each time:

// #############################################################################
// slurp all enabled feeds from the database

$feeds_result = $vbulletin->db->query_read("
	SELECT rssfeed.*, rssfeed.options AS rssoptions, user.*, forum.forumid
	FROM " . TABLE_PREFIX . "rssfeed AS rssfeed
	INNER JOIN " . TABLE_PREFIX . "user AS user ON (user.userid = rssfeed.userid)
	INNER JOIN " . TABLE_PREFIX . "forum AS forum ON (forum.forumid = rssfeed.forumid)
	WHERE rssfeed.options & " . $vbulletin->bf_misc_feedoptions['enabled'] . "
            ORDER BY lastrun ASC LIMIT 1
");

Now you don't have to make extra copies of rssposter.php, just modify the original. You would want to change it to run much more often, and if you have many feeds you may need to change the "LIMIT 1" to something higher. I guess this method has the disadvantage that you have to make sure that the task runs often enough to handle all your feeds.

[John Diver]

Thanks again Kevin!

Sorry, but I am trying to think of how this will work, for example, say I have 50 feeds (I have been adding a few over the years ) - Would

ORDER BY lastrun ASC LIMIT 1

run each of the 50 if I set it up as a cron job? (I have got the cron job working now by calling cron.php and just the RSSposter task)

Would this work if 1 of my feeds has 50 posts that haven't been retrieved, post those, then go on to the next feed?

Sorry Kevin, like I said, not very good with PHP, just trying to understand how it would work

Thank you once again!

--------------- Added [DATE]1353079405[/DATE] at [TIME]1353079405[/TIME] ---------------

I forgot to say, the reason for calling the poster by cron was because the site is just for me, so it doesn't get users that would trigger the cronimage to run the cronjob to post feeds

[kh99]

Originally Posted by John Diver

Would this work if 1 of my feeds has 50 posts that haven't been retrieved, post those, then go on to the next feed?

What I think it *should* do (without having tested it or knowing exactly what issues you're having), is process one feed every time the task runs (I suppose it will post whatever new items there are, based on the settings in the feed manager). This means if you have 50 feeds the task would need to run 50 times before getting them all. So you could either set the task to run once per minute (and set you cron job as well) and get them over the course of a hour, or increase the LIMIT to do more feeds per run. Of course this will also be affected by the "Check Feed Every..." setting in the feed manager - a feed will only be processed if enough time has passed.

I realize this still isn't the perfect solution you're looking for, but again it's something you can do with a simple edit.

[John Diver]

This is great Kevin, you have really helped Really

I am going to do this on a test forum now to see how it goes.

Once again, many thanks for helping me out Kevin