NEW Modified RSS News Feed Hack

Hack originally posted by: Xyphen - RSS News Feed Hack
Modified by: wolfstream - Modified RSS News Feed Hack

I am posting this up as a new hack because of the major changes made to how the script works. Also, I didn't want people to get confused as to which one to download since there were 3 different versions on the original post.

Xyphen released the original version, then wolfstream modified the script and corrected some errors. After that, I modified wolfstreams version to change how posts were inserted. Now the post count will increase, posts will be able to be searched, and similiar threads will work.

RSS News Feed Hack for vB 3.0.x. As long as vBulletin doesn't change the database structure much, this should be good for the next release also, and those to come.

Allows you to have a live RSS news feed (such as one from Google or CNET) in a forum. The user that posts it will basically be a bot, which *should* only be used for posting the news, but you can even make it for a regular user.

Comments are welcome! Support provided through this thread.

Latest Version:
- 1.26 - for PHP 4.3.0 & above
- 1.26a - for PHP prior to 4.3.0

vBulletin Versions:
- 3.0.0
- 3.0.1

PHP Versions:
- > 4.3.0
- < 4.3.0

Installation Overview:
- Files to edit: (2) /admincp/index.php, /includes/functions_newpost.php
- Files to upload: (4) rss.php, admincp/rss_admin.php, includes/RSS/class.RSS.php, includes/RSS/rss_update.php
- Queries to run: (4) in rss_install.php

History:
Version 1.26 (23 June 2004 - Slapyo, Natch)
- listing of bot's more intuitive (Natch)
- hopefully fixed duplicate problem (Slapyo)
- stripped html tags from title and description (Slapyo)

Version 1.25 (12 May 2004 - Slapyo)
- fix for &quot; showing up
- logs action in scheduled task log if turned on

Version 1.24 (20 April 2004 - Slapyo)
- posts now deleted with delete_thread function
- post counts updated when using delete all news

Version 1.23 (14 April 2004 - Slapyo)
- added ability to delete bots
- cleaned up includes
- fix for PHP versions prior to 4.3.0 calling html_entity_decode()

Version 1.22 (13 April 2004 - Slapyo)
- duplicate posts problem fixed
- fix for problem with ' appearing as \'

Version 1.21 (12 April 2004 - Slapyo)
- bot will not be subscribed to posts now

Version 1.2 (11 April 2004 - Slapyo)
- posts now inserted with build_new_post function

Version 1.1 (28 March 2004 - wolfstream)
- file edits taken down to 1
- cron script included

Version 1.0 (28 January 2004 - Xyphen)
- original release

Common RSS Feeds:
- Yahoo!
- CNET.com
- BBC
- Wired News
- The Register
- Ars Technica
- All Headline News

[Slapyo]

Originally Posted by SnowBot

Any news on fixing my problem yet ?

im still looking to see what i can find. so far i have no clue and the only thing i can say is that it and another mod you have are interferring.

[Slapyo]

Originally Posted by AWS

The problem with dupe posts seems to be a problem with vbulletin cron. If I run the script manually it does not post duplicates. If it runs via cron.php it runs more than once and posts duplicates. The script will run until it times out when it reaches the php timeout or until the next scheduled task is to run.
Whne I figure out a fix I'll post it.

actually it only runs one time. some people posting problems with duplicates actually have different news feeds. the script looks for the field pubDate ... some have something different for the date field like cc:date, some don't have a date field at all. also, one bug that i am pretty sure is in there that i will try and fix for the next update deals with duplicates.

if there is a " in the title, it will still post that even if it was posted before. this is because when it checks the database this is what happens:

title: this is my "news" feed
database: this is my \"news\" feed

both technically are the same, except for the escaped quotes. i think this is cause some posts with quotes in the title to be duplicated. i'm going to look into it further. i'm also trying to update the way posts are deleted with the delete all news function.

[AWS]

Beleive me it does run more than once from cron. I added some code to log when it runs and it shows it ran 4 times back to back on the scheduled time. I get 4 entries everytime it runs. For some reason it runs until the next scheduled task runs and doesn't stop after the first run.

[dethfire]

Originally Posted by AWS

Beleive me it does run more than once from cron. I added some code to log when it runs and it shows it ran 4 times back to back on the scheduled time. I get 4 entries everytime it runs. For some reason it runs until the next scheduled task runs and doesn't stop after the first run.

I agree, I think this is my problem too, however why is it running 4 times, why are the other crons ok?

[bradsears]

Slapyo - thanks very much for your continued work on this hack. You are saving us all a great deal of time.

[Slapyo]

that's odd because mine is only running 1 time. if you come up with a fix for it let me know and i will include it in the package.

could you send me the code you used to log it so i can use it in mine and see if it helps with debugging. don_lacuran@hotmail.com is my email.

there is no loop in the program to grab the rss file more than once. unless of course you have more than one rss bot. i don't have this problem on mine, but it seems to be an issue with others. hopefully we can get it worked out.

[mirdin]

Originally Posted by Slapyo

actually it only runs one time. some people posting problems with duplicates actually have different news feeds. the script looks for the field pubDate ... some have something different for the date field like cc:date, some don't have a date field at all. also, one bug that i am pretty sure is in there that i will try and fix for the next update deals with duplicates.

if there is a " in the title, it will still post that even if it was posted before. this is because when it checks the database this is what happens:

title: this is my "news" feed
database: this is my \"news\" feed

both technically are the same, except for the escaped quotes. i think this is cause some posts with quotes in the title to be duplicated. i'm going to look into it further. i'm also trying to update the way posts are deleted with the delete all news function.

Article titles like the following are not recognized as duplicates.

**** Clark Has Diabetes, Says PR Firm (AP)
$1.2M for WTC Workers (AP)

In the database the $ sign is converted to &#36 ;
I don't understand why the first one is not recognized as a duplicate.

[Mellow]

Originally Posted by Slapyo

that would be up to the feed itself if they were to accept something like that. i mean i guess you could go into the code and change the loop so that it didn't loop from:

0 -> # news articles
to
0 -> 10 (or 5)

I'd like to know this too as I only want a few news posts per day.
Joe


Would changing this:
for($j = count($rss_allItems) - 1; $j >= 0; $j--) {

to this work:
for($j = 10 - 1; $j >= 0; $j--) {

???????

Joe

[Slapyo]

Originally Posted by Mellow

I'd like to know this too as I only want a few news posts per day.
Joe


Would changing this:
for($j = count($rss_allItems) - 1; $j >= 0; $j--) {

to this work:
for($j = 10 - 1; $j >= 0; $j--) {

???????

Joe

ya, that looks like the code you would edit. then you only make your script run once every 24 hours in the cron job. and for the code you posted, you would get 10 news articles a day.

[Mellow]

Originally Posted by Slapyo

ya, that looks like the code you would edit. then you only make your script run once every 24 hours in the cron job. and for the code you posted, you would get 10 news articles a day.

Great, thanks for the quick reply

[will_lean]

hello ive just done the code edit above to try and limit the number of posts the rss bot posts but it doesnt seem to be working,i only need 10 news items posted but its posting over 100 :surprised:
any idea why the above wouldnt work?
ive set my cron to 24hr update

[Slapyo]

in rss_update.php FIND:

PHP Code:

        // insert the new items into the database
        for($j = count($rss_allItems) - 1; $j >= 0; $j--) { 


REPLACE WITH:

PHP Code:

        // insert the new items into the database
        for($j = 10 - 1; $j >= 0; $j--) { 


you are replacing "count($rss_allItems)" with the number of news articles you want. in the example code above i replaced it with 10. so it will take the 10 news articles and post them. however, i don't know if this will be a problem or not because if there are only like 5 news articles and you have hard coded in 10 news articles it might pose a problem.

[will_lean]

thanks

ive set it so i can moderate the posts from the yahoo feed so i can pick 10 feeds and prune the rest.

WICKED hack!
thanks

[Slapyo]

newest version: 1.24 (PHP > 4.3.0) and 1.24a (PHP < 4.3.0)

i rewrote the way Delete All News was done. it now uses a function within vB to delete the posts rather than a manual deletion of them. helps keep everything neat. it will also recount the posts in the forum.

[yeloc]

Is there a way not to add the news posts to the total posts number?