RSS Feeds from other sites Onto My Site, Each as a Separate Article?

[Dave234]

Hello,
I am interested in taking numerous RSS feeds from other websites and, through filters, having them go to separate addresses on my own website, each one of them displaying as an article (where people can comment, etc), and the first few sentences or paragraphs displaying as content (with a link to the full source as well).

Do you know if this is possible?

If I can't have it display as articles, then I would be fine with each appearing as a new forum post (in each individual section of my forum), but I would imagine that would be more difficult....

--------------- Added 1347598712 at 1347598712 ---------------

I have used http://pipes.yahoo.com, which actually filters feeds into individual sections.

However, how could I make this actually display on my website, each one as a separate article (where people can comment, etc) going to a predetermined section of my website based on how it is separated in pipes.yahoo.com?

Do you know what I mean?

--------------- Added 1347599498 at 1347599498 ---------------

I think that this would be a major help for a lot of people if it were possible.

I think it could be possible.

For example, I know that people can already have an RSS feed go to a specific location on their website.

Also, in pipes.yahoo.com, you can search for the text to pull from different RSS feeds in different locations of each RSS feed -- either in the title, description, author, and a couple other locations. I take this to mean that the title of the article would be in a separate location from the description, in each RSS feed. Wouldn't that mean that you could possibly direct the title to display as the title of the article on my website? And then have the description display in the body of my site's article? And then the comment section would already be there, since that is preloaded in vbulletin articles.

If there is some way to make this happen, some way to work with pipes.yahoo.com filters to make it display on my website, that would be huge. I could see a huge potential in that (for everyone's websites).

--------------- Added 1347599583 at 1347599583 ---------------

And then, based on the filters, have the results of one filter go to one article section of my website, and have the results of another filter go to another article section of my website, etc.

Does that make sense?

[kh99]

If I'm understanding you, I think you could do what you want using threads posted in forums without any changes. You'd just have to use the feed manager to set up a different feed for each forum (I'm assuming the yahoo pipes thing lets you construct a feed url that provides the articles filtered in some way).

As for making them articles, I have to admit I'm not as familiar with the CMS as I probably should be, but I think articles are nasically threads, so that it might only take relatively minor changes to the rss poster code to make articles instead of threads.

[Dave234]

kh99, that would be awesome if I could have each RSS feed news story post as a separate thread.

You are referring to the feed manager in the admincp, correct?

Would the feed manager allow me to use the Item Description as the body of the forum post, and the Item Title as the title of the forum post?

If you can help me out with this, I would be eternally grateful, as this would be truly awesome and a major help.

[kh99]

Have you looked at the "RSS Feed Manager"? I think a lot of it is self-explanatory. There's a "Post Items as Threads" section that is selected by default, and above that is a Templates section where you might be able to rearrange what's used for the title and body of the post. You need to enter a user name and select a forum (I'm going bottom to top for some reason). If there's anything you can't figure out you can always ask here.

[Dave234]

Do you know if there is a way to make "Days for Announcement to Remain Active" to be indefinite? I don't want these posts to disappear from my forum!

--------------- Added 1347690983 at 1347690983 ---------------

Do you know if I can copy any of the RSS Feeds that I created in my Feed Manager? It would make it so much easier if I could just copy them so that I can use the same info on the copies but with another RSS feed URL...

[kh99]

Do you know if there is a way to make "Days for Announcement to Remain Active" to be indefinite? I don't want these posts to disappear from my forum!

"Days for Announcement to Remain Active" only applies if you choose to post the feed as announcements. The last two sections on the "Add New Feed" page are for adding the items as threads or adding them as announcements, and only one can be selected at a time (there are radio buttons in the header next to the section titles). So if you're posting items as threads, you don't have to worry about that setting, the threads will never disapper.


Do you know if I can copy any of the RSS Feeds that I created in my Feed Manager? It would make it so much easier if I could just copy them so that I can use the same info on the copies but with another RSS feed URL...

There is no feature to do that, but you might be able to do it directly from the database by writing directly to the rssfeed table.

[Dave234]

Can I write directly to the rssfeed table within the admincp?

How exactly would I do that? I definitely don't want to mess with anything on the ftp.

--------------- Added 1347722271 at 1347722271 ---------------

Also do you know how I can bring any image or video from the article into my forum post, along with the description and link?

The feed template in my admincp Feed Manager is the standard

{feed:description}

More... ({feed:link})

[kh99]

Changing the database directly is probably an advanced thing. In short, each row in the rssfeed table has the data for one feed, so if you wanted a lot of similar feeds you might be able to do it by duplicating rows. But unless you know at least some simple sql, it's probably better for you to use the RSS Feed Manager.

As for your other question, I would have to look in to it but I won't have a chance to do it today.

[Dave234]

Would I do this in the admincp > Maintenance > Execute SQL Query ?

What would be the next step?

Thank you very much for helping me with this.

[kh99]

Right, you would do it in Execute SQL Query, and I guess you could do this (to duplicate one row):
INSERT INTO rssfeed
(SELECT NULL, `title`, `url`, `port`, `ttl`, `maxresults`, `userid`, `forumid`, `prefixid`, `iconid`, `titletemplate`, `bodytemplate`, `searchwords`, `itemtype`, `threadactiondelay`, `endannouncement`, `options`, 0 FROM rssfeed WHERE rssfeedid = 1)


And you'd change the 1 in "rssfeedid = 1" (at the end0 to the feed id that you want to copy. (So I guess it's not really that advanced :) ). You may want to go to the Scheduled Task Manager first and temporarily disable the RSS Poster Robot task, or else it may run before you get a chance to go in and edit your new feeds.

[Dave234]

I wish I knew enough about this stuff to understand what you just wrote......

A couple questions:

1. Where would I find my feed id?
2. Can you please go step by step (I mean like baby step by baby step) of what I need to do after clicking on admincp > Maintenance > Execute SQL Query? I see three boxes where I can type stuff. The top box says "Automatic Query". Beneath that it says "Manual Query". The one beneath that says "Results to Show Per Page" and it is default set at 20.

By the way, the RSS feed that is not working for me with pulling the video or images into my posts is: http://feeds.bbci.co.uk/news/world/rss.xml

For example, on this link http://www.bbc.co.uk/news/world-asia-china-19613295# it is only showing this in my forum post:

VIDEO: China holds world goldfish championships

More than 3,000 goldfish have been competing at the first International Goldfish Championships in China's Fujian Province.

More...

[Dave234]

Also, my RSS feeds don't seem to be updating..... I have to go into admincp RSS Feed Manager and click "Run" for the feeds, but the past couple times I tried it today it goes to one of those "Internet Explorer cannot display the webpage" screens (like that screen within the frame of the admincp, where my internet is working fine, but the message dispalys anyway).

This has happened quite a few times before, but eventually it would work like if I closed out of the screen and then reopened it. However, the last couple times I tried it it simply would not work.

I have the RSS feeds being updated every 30 minutes, so it really should be updating the website with new RSS feeds...........

Do you know why this might be happening?

[kh99]

I wish I knew enough about this stuff to understand what you just wrote......

Well, you might want to consider just creating the feeds through the Manager, unless you have hundreds you want to change. Changing the database directly isn't really a good idea in general. You should be pretty safe only making changes ot the rssfeed table, but it's always a good idea to have a backup of your database.


A couple questions:

1. Where would I find my feed id?

It doesn't display directly, but it's part of the url when you edit a feed, so you can see it in the url when you mouse over the Edit link in some browsers, or else you can check the "Frame Info" (in FireFox, for instance). Or you can use a query, like:
SELECT title, rssfeedid FROM rssfeed


and it will display the titles and feed ids of all your feeds.


2. Can you please go step by step (I mean like baby step by baby step) of what I need to do after clicking on admincp > Maintenance > Execute SQL Query?

It's easy, you just enter your query into the "Manual Query" box and hit continue. If you're doing a query that changes the database it will warn you on the next screen and you'll have to hit continue again. That's it.

[kh99]

Also, my RSS feeds don't seem to be updating.....

Is your forum busy? The feeds update throgh the Scheduled Tasks mechanism, and that only runs if your forum has traffic.


Do you know why this might be happening?

I don't know. I think that error probably means that the php script is ending in an error, but unfortunately that doesn't narrow it down very much. If you've been changing the feed "template" areas like we were talking about above, maybe there's an error in one of those or something (but that's just a wild guess).

[Dave234]

Well, you might want to consider just creating the feeds through the Manager, unless you have hundreds you want to change. Changing the database directly isn't really a good idea in general. You should be pretty safe only making changes ot the rssfeed table, but it's always a good idea to have a backup of your database.

I think, just to be safe, I will enter the rss feeds one at a time, in my RSS Manager, even though there are literally hundreds. I really don't want to mess with the database.

Is your forum busy? The feeds update throgh the Scheduled Tasks mechanism, and that only runs if your forum has traffic.

There is no one on my forum yet -- it's closed at the moment. I'm trying to get it up and running before I open it to the public.

I don't know. I think that error probably means that the php script is ending in an error, but unfortunately that doesn't narrow it down very much. If you've been changing the feed "template" areas like we were talking about above, maybe there's an error in one of those or something (but that's just a wild guess).

Maybe it's because I'm doing the rss feeds through yahoo pipes. I think I'm going to just have to enter them into the RSS Manager one by one, even though that is going to be literally hundreds of them. If it's gotta be done that way, it's gotta be done that way.

Would you be able to provide any insight on why the RSS feed that is not working for me with pulling the video or images into my posts?

Using the RSS feed, for example, of: http://feeds.bbci.co.uk/news/world/rss.xml

For example, on this link http://www.bbc.co.uk/news/world-asia-china-19613295# it is only showing this in my forum post:

VIDEO: China holds world goldfish championships

More than 3,000 goldfish have been competing at the first International Goldfish Championships in China's Fujian Province.

More...

--------------- Added 1347883800 at 1347883800 ---------------

I don't know why it's STILL not working. I've unchecked the vast majority of the RSS feeds and then clicked "Run Scheduled Task Now", and it still comes to that "Internet Explorer cannot display the webpage". Why would it be doing this?????

--------------- Added 1347883848 at 1347883848 ---------------

This is happening EVERY time now.

--------------- Added 1347883994 at 1347883994 ---------------

I don't know. I think that error probably means that the php script is ending in an error, but unfortunately that doesn't narrow it down very much. If you've been changing the feed "template" areas like we were talking about above, maybe there's an error in one of those or something (but that's just a wild guess).

I definitely haven't been doing that. I thought it might have something to do with yahoo pipes but even after I unchecked the ones with the yahoo pipes rss feeds, it still shows the "Internet Explorer cannot display the webpage" error....................... It used to just do this every once in a while, but now it is literally every time, so much so that I can't even run the rss feeds now. I really need this to work.

--------------- Added 1347941693 at 1347941693 ---------------

Please, can someone help me with this?

[Dave234]

Anybody?

[kh99]

Do you have access to your web server error logs? You might be able to look there and see if there is any more information. Otherwise, I suppose it might be some problem with one feed so you might try disabling feeds to see if you can get any of them to work. Or if you don't have too many feeds defined you might try deleting them all and starting over.

[Dave234]

Could there be too many feeds, making it not work?

--------------- Added 1348169654 at 1348169654 ---------------

I've asked vbulletin.com to assist, and they will help, but they say the following:

- Whether or not you have edited any files (if you have installed hacks or modified any of the vBulletin PHP files, we cannot offer any support until the original files are restored.)

Which makes me very nervous. I would much prefer to fix it on my own than have them mess with anything I've spent a lot of time already doing.

--------------- Added 1348171993 at 1348171993 ---------------

To disable feeds, I would just need to uncheck all of the feeds in my RSS Manager?

[kh99]

Yeah, it's too much work for them to have to debug problems that may be caused by mods. Have you made edits to any of the vb files?

Yes, I meant just uncheck feeds in the manager to disable them.

One thing you might try: edit the file includes/cron/rssposter.php, and at the top near the error_reporting line, add this:
ini_set("display_errors", "1");


That may show the error when you run the scheduled task manually. (remove that line when you're done).

[Dave234]

I don't know, I really don't want to mess with anything in the ftp....... Can I do this in admincp?

--------------- Added 1348239616 at 1348239616 ---------------

What could be wrong with it, if it was working before? Is there anything within the settings that say I can only have a maximum number of RSS feeds at one time?

[kh99]

I don't really know why it might have been working then stopped. You didn't make any changes? It was you who was thinking about creating more feeds by duplicating the rows in the database, wasn't it?

I suppose it could have something to do with the content of one of the feeds, but that's just a guess.

You can't make that change I mentioned via the admincp, so if you don't want to ftp anything I'm out of ideas. You could try truncating the rsslog table (since I can't think of anything else to tell you), or delete all your feeds and re-enter them, but if you do that you may end up posting duplicates of any items that are currently in the feeds (since it uses the rsslog table to keep track of what's been posted already).

To be honest you may need someone with access to your site who can go in and debug it and see what's wrong, but you'd probably have to pay someone to do that.

[Dave234]

I think it must be the content of one of my feeds.

I didn't change anything in the database.

I definitely don't want to mess with the ftp.

I will switch off all of the rss feeds when I get back home, and see what happens.

--------------- Added 1348243602 at 1348243602 ---------------

In between entering the RSS feeds and there being this problem arising, I didn't do anything else to my website.... Does that mean anything? It isn't like once I reach a certain number of RSS feeds there would suddenly be a problem, right?

[kh99]

I don't know any reason why there would be a problem just because of the number of feeds, but if you have a large number or some are slow I guess something could be timing out at some point.

[Dave234]

That "Internet Explorer cannot display the webpage" -- that doesn't sound like an error that would occur for anything other than timing out, right? And timing out wouldn't be caused by anything other than just a faulty RSS feed, right?

[kh99]

I don't know. I think "cannot display the page" could be other things, and a timeout could happen in different situations (I think there's a php setting that sets the maximum number of seconds a php script can be running before it times out).

If you run the rss poster task from the admincp, how long does it take before you get an error? Is it quick, or does it take maybe 30 seconds?

[Dave234]

It takes about 30 seconds, maybe more. Actually 30 seconds sounds about right.

--------------- Added 1348258921 at 1348258921 ---------------

What does that mean?

[kh99]

Well, I was thinking that it may mean that the script is timing out, since there's a maximum time that a php script is allowed to run. But I just looked at the rssposter.php code and it already attempts to turn off the time limit. But apparently this won't work if your php is set up in "safe mode", and in that case, according to the php docs, there's nothing you can do to increase it.

Try going to Maintenance > View PHP Info and look at the values for safe_mode and max_execution_time.

[Dave234]

I went into Maintenance > View PHP Info, and saw the following:

safe_mode: Local Value = Off Master Value = Off
safe_mode_exec_dir: Local Value = no value Master Value = no value
safe_mode_grid: Local Value = Off Master Value = Off
safe_mode_include_dir: Local Value = no value Master Value = no value

max_execution_time: Local Value = 30 Master Value = 30

--------------- Added 1348266099 at 1348266099 ---------------

One of my feeds went through. It was one feed......it should have been much more. And it didn't go to the right subforum.....

--------------- Added 1348266203 at 1348266203 ---------------

And actually I just timed how long it takes from me pressing "Run Scheduled Task Now" in the RSS Feed Manager and it going to the "Internet Explorer cannot display the webpage". It is 8 seconds. Every time.

--------------- Added 1348280797 at 1348280797 ---------------

Does any of that mean anything?

--------------- Added 1348286310 at 1348286310 ---------------

It seems to be when I have a combination of RSS feeds checked off at the same time. It's weird because a lot of them work one at a time.

Do you know why this could be?

[Dave234]

Anybody?