RSS News Feed Hack

My friend made this hack, he owns http://www.liquidpro.net. Check him out.
You may contact him at liquidpro@neo.rr.com. Or you can IM him on AIM at LiquidPro10.

An example is shown at http://www.xtreme-forumz.net. Go to the Techno forum under the General category. Then the CNET News sub-forum.

RSS News Feed Hack for vB 3.0.0. As long as vBulletin doesn't change the database structure much, this should be good for the next release also, and those to come.

Allows you to have a live RSS news feed (such as one from Google or CNET) in a forum. The user that posts it will basically be a bot, which *should* only be used for posting the news, but you can even make it for a regular user.

Comments are welcome!

Instructons:
Note: DON'T FORGET TO MAKE A BACKUP OF EACH FILE BEFORE YOU MODIFY IT!

Copy the files to your web server... keep the same directory structure as in the zip file.
Run the rss_install.php file to setup the RSS script.
If you're upgrading run the rss_upgrade.php file to setup the RSS script. Don't forget to remove all previous modifications (you can find them by searching for "LiquidPro")

#### open the /index.php script ####
Find this:

PHP Code:

require_once('./includes/functions_forumlist.php'); 


Add below it:

PHP Code:

require_once('./includes/RSS/rss_update.php'); 


#### open the /forumdisplay.php script ####
Find this:

PHP Code:

require_once('./includes/functions_forumdisplay.php'); 


Add below it:

PHP Code:

require_once('./includes/RSS/rss_update.php'); 


#### open the /admincp/index.php script ####

Find this:

PHP Code:

construct_nav_group("$vbphrase[styles] &amp; $vbphrase[templates]", '<hr />'); 


Add below it:

PHP Code:

// Manage Hacks
// by: Andrew Wickham
// LiquidPro Inc.
//
construct_nav_option('RSS News Feed', 'rss_admin.php', '<br />');
construct_nav_group('Hacks');
//
// End Manage Hacks 


You're done, thanks for installing the hack.

Hack by: Andrew Wickham (http://www.liquidpro.net)
Hack for: Xtreme-Forumz (http://www.xtreme-forumz.net)

[sjau]

Btw,

I'm just studying your code now and I just made the comparison between old entries and new RSS items visible by having the array values displayed.

Array display function:

PHP Code:

function displayArray($aArray) {
    if (is_array($aArray) && (count($aArray) > 0)) {
        print("<table border=1>");
        print("<tr><th>Key</th><th>Value</th></tr>");
        foreach ($aArray as $aKey => $aValue) {
            print("<tr>");
            if (!is_array($aValue)) {
                if (empty($aValue)) {
                    print("<td>$aKey</td><td><i>$aValue</i></td>");
                } else {
                    print("<td>$aKey</td><td>$aValue</td>");
                }
            } else {
                print("<td>$aKey(array)</td><td>");
                displayArray($aValue);
                print("</td>");
            }
            print("</tr>");
        }
        print("</table>");
    } else {
        print("<i>empty or invalid</i>");
    }
} 


Altered code:

PHP Code:

        // if there's no items in the database
        if($DB_site->num_rows($rss_query) == 0) {
            for($i = count($rss_allItems) - 1; $i >= 0; $i--) {
                $rss_allItems[$i][OLD] = false;
            }
        }

        displayArray($rss_allItems);

        // filter out the old items

        $rss_thread_data = $DB_site->fetch_array($rss_query);

        while($rss_thread_data = $DB_site->fetch_array($rss_query)) {
            for($j = count($rss_allItems) - 1; $j >= 0; $j--) {
                $rss_thread_data2[$j][title] = $rss_thread_data[title];
                $rss_thread_data2[$j][rss_date] = $rss_thread_data[rss_date];
            }
        }

        echo "<br><br>";

        displayArray($rss_thread_data2); 


And the rest is all commented out.

The result I'm getting is that:

- No date is set

it seems that the database still should be filled...
Looking more into this matter

http://wsjb78.com/forums/

Edit: I know my code is not quite right yet to display them accordingly



Actually forget all of the above settings. I found the problem:

Change this in your /forum/includes/RSS/rss_update.php:

PHP Code:

        for($i = count($rss_allItems) - 1; $i >= 0; $i--) {
            $rss_allItems[$i][OLD] = true;
        } 


To that:

PHP Code:

        for($i = count($rss_allItems) - 1; $i >= 0; $i--) {
            $rss_allItems[$i][OLD] = false;
        } 


[KuraFire]

Originally Posted by sjau

I'm just playing now with the idea of using a vB cron for periodic update of the feeds. I tend to think that on larger boards it's not a good idea to query the RSS file like ever 10-20 seconds just when someone goes in the according forum. I guess you could end up your IP being blocked!

Yup, many sites, especially larger ones, will block your IP (some do it for an hour, some for a day, some for a week, others permanently until you contact them) if you make more than one request per 30 / 60 minutes.

So having it fetch the XML/RSS feeds every time the forum page is loaded is Really likely to get your server ip banned from that xml/rss feed. And then you'll be complaining here that it doesn't work and cause insanity for poor Xyphen who's going through his code to see why that is

[sjau]

I think I've found the problem. Posted it into my previous post (didn't see KuraFire did already add another post).

[sjau]

Ok, I've tested this now several times and my modification seems to work without problems.

Further more I did remove the require code from the index.php and forumdisplay.php and copied the supplied two rss files from ./includes/RSS to ./includes/cron and setup a cron for them.
It seems that works also.

[Xyphen]

Yes, that definitely seemed to be a problem... in v2.0 there's a completely new RSS class, so that problem is definitely fixed. I see what you mean about the CRON job and such, I will look into it.

[KuraFire]

Cron job is much safer and nicer to do this with

[NuclioN]

Is there a way to pull the news from RSS into the postbit itself when you click a link? I've seen it before but i don't know how they do that.

[sjau]

Xyphen:

I have another three things in mind:

1.) Add in the admin section a name to each feed so that you only have to create one user-bot but you can distinguish them by the name given in the AdminCP and the name given in the AdminCP should then also be used as username when wwriting data to mysql.

e.g. I currently pull data from NZZ Online, from NZZ Wirtschaft, 20min Ausland, AltaVista and Google.de

I currently have for each news service another user setup but I think I should be able to do it with one user only that can be distinguished in the admin and forum section.

2.) You use in in code "<br><br>" do make row breaks. Maybe you could check whether on this board HTML is enabled and if not use "\n\n" instead and "View full article" could be a phrase so that it can be translated easily into other languages.

3.) Increase forum counts when news are added and decrease forum counts when they are deleted.
Maybe adding also a feature that deletes news older than xx days.

Anyway, great work so far and I would help you out but my skills aren't as advanced as yours although I'm learning.

[wolfstream]

  for($i = count($rss_allItems) - 1; $i >= 0; $i--) {
            $rss_allItems[$i][OLD] = false;
        }

That fixed the problem right up, indeed.

Now, there are a couple more things that are issues. For one there's no check for mysql_escape_string which will be bad if you actually grab rss feeds from, say a site that uses " or '. I've already run into this one tonight;
Fix:
In includes/RSS/rss_update.php, look for

$rss_title = str_replace("'", "\'", $rss_title);
$rss_description = str_replace("'", "\'", $rss_description);

and change it to:

         $rss_description=mysql_escape_string($rss_description);
         $rss_title=mysql_escape_string($rss_title);
//         $rss_title = str_replace("'", "\'", $rss_title);
//        $rss_description = str_replace("'", "\'", $rss_description);

Essentially, this is the same thing, but for some reason the replace &apos; doesn't work as it should, in the longrun. I noticed this when I got feedback from the next script which I'll post:

In response to the cron thing, it's really a simple thing to do (until this is officially added to the cron somehow). Create a file in your public_html, call it rssfeed.php (or whatever), and in it, just add:

<?
require_once('./global.php');
require_once('./includes/functions_bigthree.php');
require_once('./includes/functions_forumlist.php');
require_once('./includes/RSS/rss_update.php');
?>

and for a cron entry, just call

*/5 * * * * php /path/to/your/file/rssfeed.php

This will update the news feeds every 5 minutes, rather than every time someone visits the forum, which would be more efficient. 5 minutes may (still) be too much, but I doubt it.

BTW, i've integrated this into vbadvanced, for those wanting info on how, let me know and if there's enough interest I'll post it here.

As far as forums and counts, I agree this is something that shouldn't be updated. However, if you (really) want to update your forums and posts, having this count in them, you can update your postcounts from admincp. This will fix that problem. Hopefully, I'll be able to come up with a workaround for that in the near future, but who knows.

Great code, glad to see it finally working!!

[wsjb78]

I'd be interested to know how you did that integrate into vbAdvanced. I've installed now vbIndex 3.0.x and also vbAdvanced but I only have troubles with certain things in vbIndex so I assumbe I'll stick to vBadvanced.

Btw, my code change (true --> false) was quite simple if you look at the logic of what is approved and not. I was looking first at the wrong spot either but basically if you set it to "true" it marks all fetched news items from the feed as [old], then if there are no items in the db it markes them all as [new], and then it fetches the items from the database and marks duplicate entries as [old] (which they already were).

Regarding the Cron:
How does it work? Those vB crons aren't really added to the system crontab right?

Does it check everytime when a vB page is called whether one of the specified crons if overdue and then execute it?

[wolfstream]

I'll get together tonight or in the next couple of days and put a hack together. It certainly wasn't a 3 or 4 line edit and I'm almost positive I missed something in there. Once I'm sure it works, I'll post it.

As far as cron, it's not added to the user's crontab (vbulletin isn't), no. In fact, I'd bet you'd run into the same problem there too, because it is called every time the page updates, so your best answer would be to create the file and call it via cron however often you want it to. I'm only a beginning hack however (as far as vb is concerned), haven't played with the cron stuff too much (yet).

[sjau]

you're code works fine

[mcyates]

Where can i find news feeds for soccer/football

[mcyates]

When the news is updating i always get this error. Obviously with a different storyline.

Database error in vBulletin 3.0.0 Release Candidate 4:

Invalid SQL: INSERT INTO post (`threadid`, `username`, `userid`, `title`, `dateline`, `pagetext`, `allowsmilie`, `showsignature`, `visible`) VALUES ('21596', 'News Poster', '3137', 'Stevenage 0-1 Telford', '1076235015', 'Stevenage's hopes of closing the gap on the play-off places suffered a blip as they crashed 1-0 at home to Telford.<br><br>View the Entire Article
', '1', '1', '1')
mysql error: You have an error in your SQL syntax. Check the manual that corresponds to your MySQL server version for the right syntax to use near 's hopes of closing the gap on the play-off places suffered a bl

mysql error number: 1064

Date: Sunday 08th of February 2004 05:10:18 AM
Script: http://www.myfootballforum.com/forumdisplay.php?f=318
Referer: http://www.myfootballforum.com/index.php?
Username: boro_boy
IP Address: 62.254.64.9

[mcyates]

now my sites, stopped working. I'm running RC4