news2vb - Moreover news headlines to vBulletin discussion thread

[Cloud-Warrior]

Hope the instructions at the top are clear, if not please post here.

To install this utility, copy the single PHP file to your vB dir. Edit it and change the password to whatever you wish. Change the yourvbdir variable to the URL pointing to your vB directory.

As demo - here are some sample output threads:

http://www.boards.ie/vbulletin/showthread.php?s=&threadid=44475
http://www.boards.ie/vbulletin/showthread.php?s=&threadid=44522

Also, have a look at the menu here:

http://www.boards.ie/vbulletin/news2vb.php?jaction=picksource

This script will generate messages of the following form in whatever forum you wish, using any of the Moreover newsfeeds.


Microsoft s Xbox Fails To Wow Japanese Market (http://c.moreover.com/click/here.pl?t33811472)
(osOpnion, Mar 13 2002 8:17AM) Discuss (http://www.boards.ie/vbulletin/newreply.php?action=newreply&threadid=44475&title=Microsoft+s+Xbox+Fails+To+Wow+Japanese+Marke t)
Xbox - dont believe the hype (http://c.moreover.com/click/here.pl?t33791692)
(IT-Director.com, Mar 13 2002 1:33AM) Discuss (http://www.boards.ie/vbulletin/newreply.php?action=newreply&threadid=44475&title=Xbox+-+dont+believe+the+hype)


Changelog

1.1 - "Post News" output now shows BB code parsed message, URL to thread
1.1 - Fixed missing references to $yourvburl

[Cloud-Warrior]

So what does the hack do - basically if you call the PHP script with the jaction=picksource command you can see a list of Moreover feeds (www.moreover.com) and a list of your forums, and you can post the headlines into the forum of your choice (as long as you supply the correct password as hardcoded into the script).

Alternatively, you can ignore the categories and perform a custom list of headlines (by searching in the headline keywords or a query through the article text - keyword and query boxes respectively), or you can enter a feed number in the feed box (some unedited Moreover feeds don't have category names - you can see the feed ID number in examples at their site).

You should also be able to generate forum posts from a script, e.g. using wget you would do something like:


set url = http://www.wherever.com/vbulletin/news2vb.php
wget -O /dev/null "$url?jaction=postnews&jpassword=cloudbot&jcategory=Arts%2520and%2520culture%2520news&jforum=21"
wget -O /dev/null "$url?jaction=postnews&jpassword=cloudbot&jfeed=15771&jtitle=Buffy+news&jforum=93"
wget -O /dev/null "$url?jaction=postnews&jpassword=cloudbot&jkeywords=lord+of+the+rings&jforum=103"
wget -O /dev/null "$url?jaction=postnews&jpassword=cloudbot&jquery=%22babylon+5%22&jforum=93"


Sample screen from the menu is attached.

Any questions, please ask.

[Psychdrone]

got a demo for us!

[Erwin]

It works!!!

I am making it a cron job and running the query every 24 hours... thanks! :)

[Erwin]

Cool - the only thing that stumps me is the

$jiconid = "1";

variable - I put in a number 1, but the icon does not show up.

Am I meant to put in the full gif file name, or the whole path to the icon file?

Thanks for your help...

[Erwin]

I see... the posticon shows up in the actual showthread itself, but not in forumdisplay... I wonder why...

[Cloud-Warrior]

What you should put in for that post icon variable ($iconid) is the ID number of the icon in the icons table (can check in PHPMyAdmin or else in the CP).

I've moved the demo links up top.

[AndyTSJ]

I like this, but would love to know how it could be automatically programmed to run to post a set of stories each day :)

[Cloud-Warrior]

If you are on Unix you can use crontab to schedule the wget (as in the quoted example earlier in this thread) command for a particular feed.

Other options for Windows that I can think of off the top of my head would be to install Lynx and use the Windows scheduler to call the URL corresponding to a particular feed from Lynx - you'll see when you use the menu version to post a particular feed that there is a URL in the output - that is the URL you would call from an automated script (on Unix or Windows). I'm not really that familiar with any other options for calling a URL in Windows, I guess you could also just write a PHP or Perl script to download the URLs and schedule that daily with Windows scheduler as well.

Here's the syntax for the URLs (password hardcoded into the script is cloudbot):


This one gets headlines in the category "Arts and Culture" and posts them to forum ID number 1:
http://www.whereever.com/vbulletin/news2vb.php?jaction=postnews&jpassword=cloudbot&jcategory=Arts%2520and%2520culture%2520news&jforum=1
This uses one of the unedited feeds which are identified by a feed number (see feeds on the left of this page (http://www.moreover.com/cgi-local/page?o=portal&feed=500)):
http://www.whereever.com/vbulletin/news2vb.php?jaction=postnews&jpassword=cloudbot&jfeed=15771&jtitle=Buffy+news&jforum=1
This syntax searches through all of today's headline keywords and returns headlines matching those keywords:
http://www.whereever.com/vbulletin/news2vb.php?jaction=postnews&jpassword=cloudbot&jkeywords=lord+of+the+rings&jforum=1
This query finds matches in article text for the past few weeks:
http://www.whereever.com/vbulletin/news2vb.php?jaction=postnews&jpassword=cloudbot&jquery=%22babylon+5%22&jforum=1


Again I'd refer to http://w.moreover.com/dev/ and http://w.moreover.com/dev/custom/ for help on Moreover feeds.

Let me know if I can help further, or if there are any bugs/improvements you've spotted.

John.
--

[Erwin]

Originally posted by Cloud-Warrior
What you should put in for that post icon variable ($iconid) is the ID number of the icon in the icons table (can check in PHPMyAdmin or else in the CP).


Well, I did that. Like I said, the post icon shows up in the thread, but in the forum display it is a broken image, with the URL being just the forum directory rather than the URL to the image file.

Strange...

[Cloud-Warrior]

Weird, seems to work for my forum - what's your URL?

[Cloud-Warrior]

Yeah it's strange that the icon is not being shown properly in your threadlisting Erwin, did you use an ID number that is shown via PHPMyAdmin - it may not be the same as the icon name - e.g. my iconid 37 corresponds to icon14.gif.

No icon is being set, I guess to solve the broken image at least you can set the iconid to 0 (none will appear). This is what is being shown in your threadlisting

<img src="" alt="" width="15" height="15" border="0">

Word Wrap Bug

Another thing to note is that you may notice a break occuring somewhere in the middle of the Discuss URL or the Discuss text - you will either have to set your Word Wrap to a large value or else to 0 to stop this.

[Erwin]

Originally posted by Cloud-Warrior
No icon is being set, I guess to solve the broken image at least you can set the iconid to 0 (none will appear). This is what is being shown in your threadlisting


Changing iconid to "0" fixed it. The default icon shows up now which is what I want anyway!



Another thing to note is that you may notice a break occuring somewhere in the middle of the Discuss URL or the Discuss text - you will either have to set your Word Wrap to a large value or else to 0 to stop this. [/B]


I changed Word Wrap to "0" and it fixed that problem too.


Thanks!


I am now trying to get my crontab to work - it is not working for some reason.

This is my cron command line:


0 0 * * * wget -O /dev/null "http://www.mysite.com/forum/news2vb.php?jaction=postnews&jpassword=mypassword&jcategory=Whatever%2520news&jforum=1&jtitle=My+News"


I would expect it to run every 24 hours at midnight server time, but it hasn't for some reason. (I have replaced the site, password and forum details with something else for obvious reasons).

Any ideas?

[Cloud-Warrior]

I can't see anything wrong with that crontab, maybe try changing it to 59 23 to see what happens. I noticed your posts are occuring at 8.25 which doesn't match up to midnight by any stretch of the imagination...

[Erwin]

No, I am doing those news inserts manually... the crontab ain't working! :(

I'll try your suggestion...

Is there another way I can set up crontab to do this?

[Cloud-Warrior]

Maybe you need to specify the full path to wget... Try a whereis wget. Alternatively you can use lynx if it is installed, in your crontab you'd probably do something like (specifying the full path to lynx as well):

0 7 * * * TERM=vt100 /path/to/lynx -source "http://..." >> /dev/null

I've added one or two things to the Post News output, modified file at the top.

[Erwin]

Aha!!!

I have lynx, and I use that to run my USENET crontab, so there you go, this is going to work - I did try it with yours, but I did not put the TERM variable in.

Will let you know if it does work.

Downloaded and using the new news2vb.php file - the output is heaps better! Thanks!

Just one small bug:

You have $yourvbdir at the top but $yourvburl in the code.

Change the $vbdir to $vburl or vice versa for the "Discuss" links to work. :)

Thanks heaps!

[Cloud-Warrior]

Actually I think it should now say yourvburl in both, I'm guessing it was yourvbdir in version 1.0. Either way, glad it's working - time to move to Full Hacks I think.

[Wayne Luke]

I installed this and plan on making a few changes to make it more robust and more suited to my needs. Thanks.

[Cloud-Warrior]

Thanks Wayne, if you think any of the changes should be implemented for all installers, please let me know.

[Erwin]

Hi,

I've tried variations on the crontab, and just can't get the thing to be automated!

Any ideas? I've tried:

* * * * * TERM=vt100 /path/to/lynx -source "http://..." >> /dev/null

* * * * * /path/to/lynx -source "http://..." >> /dev/null

* * * * * /path/to/lynx -source "http://..."

Ive tried the curl command, lynx -dump, even cgi php commands - no luck! :)

My question is - the php file has variables outside of the file ie. ?jaction= etc. does that work with cron or not?

Would the path to wget be the same as the path to lynx?

[Erwin]

Might have to place all the variables inside the php file, and use lynx - source -> having variables outside might not work for cron.

If anyone out there can automate this with cron, please post your command line, thanks!

[Cloud-Warrior]

The paths could be the same, but possibly different, you'll have to do a whereis lynx as well possibly.

I've done lynx crontab commands with parameters and they usually work fine. Maybe try putting the wget or lynx lines in a .sh file and call that via cron.

[Erwin]

I'll keep trying... like I said, any unix cron experts out there, please feel free to make suggestions! :)

[Erwin]

I think this one works for me...

0 0 * * * /path/to/lynx -source "http://..." >> /dev/null

:)

Now to waith 24 hours to make sure it repeats itself!!! :D

[Cloud-Warrior]

Yeah I guess having 5 stars could be a problem :-)

[Erwin]

Actually, it didn't work. Had some discussion with my host's support people, and we are finally trying it out as an external post.sh script - the sugestion from them is that the command line may be too long. I have no idea why it would work as a manual command, but not as an automatic cron job.

[Visionray]

how do you limit the number of news feeds the script posts?

ALso, every title of every news feed is dated at 2/12/02. Whats up with that?

[Cloud-Warrior]

That would be yesterday's date :-) The date shown is the date the script is run (the post date).

If you want to limit the number of articles (I think this is what you mean), you'll have to hack the source and add in a counter somewhere in or around the loop if ($jitemheadline != "") { }. Or you may be able to specify the number of articles you want from moreover somehow (can't remember if there's an option or not, check http://w.moreover.com/dev/).

John.
--

[Visionray]

LOL sorry man. When it said 2-12-02 I thought it was talking about Februrary 12th, 2002....me being from America and our Calendar system is backwards. Thanks ;) :D

[fimfimfim]

First off, very cool mod / hack! :D

I was wodnering if there was an easy way to make each headline a thread with a link to the story vs putting all the days news into a thread as posts. Any help is appreciated. TIA!

[Cloud-Warrior]

Do you mean, instead of a thread, you'd create a pseudo-thread that would actually link to the news story?

That'd involve some hacking of vB I think, more than just an add-on :)

Or do you just mean having a new thread for each news item, with the subject = headline and posttext = link to article?

That'd be easy enough to do I'd say, in the loop where it says:

if ($jitemheadline != "") {
...
}

you'd post a new thread and post for each headline instead of one big thread and post with all headlines after the loop.

John.
--

[fimfimfim]

The latter is what I was looking for. I'll give it a shot. Thanks again!

[fimfimfim]

Got it! Thanks for the input. Had a problem for a while where I wasn't setting $jmessage to null after each insert.. duh :rolleyes: Anyway, its working great. Once again thanks for the mod and your help!

[limey]

Can you post how you did the new thread per story listing?

Aside from queries and keywords, can you limit the number of stories pulled?

[fimfimfim]

Sure here's what I did:

In the section that Could-Warrior mentioned:

if ($jitemheadline != "") {

I added this to limit the headlines to within one hour of the script executing:

$currenttime=time();
$articletime=strtotime($jitemextractiontime);
if ($articletime < ($currenttime - 3600))
continue;

I added this to limit the duplicate headlines:

$result=$DB_site->query("SELECT threadid FROM thread WHERE title LIKE '".addslashes($jitemheadline)."' ");
if ($DB_site->num_rows($result) > 0)
continue;

Lastly, I moved all of this: (which is all Cloud-Warrior's original code with a small change to the VALUES list to make the title of the article appear as the thread subject)

else {
$jmessage .= " ".$jitemheadline." ($jitemarticleurl)\n";
$jmessage .= "($jitemsource, $jitemextractiontime) \n";
$jsubject = urlencode($jitemheadline);
$jmessage = ereg_replace("INSERTTITLE", "$jsubject", $jmessage);
$DB_site->query("INSERT INTO thread (threadid,title,lastpost,forumid,open,replycount,p ostusername,postuserid,lastposter,dateline,iconid, visible,attach) VALUES (NULL,'".addslashes($jitemheadline)."','".time()."','$jforum','1','0','".addslashes($jfromwho)."','0','".addslashes($jfromwho)."','".time()."','$jiconid','1','0')");
$jthreadid=$DB_site->insert_id();
$jmessage = ereg_replace("INSERTTHREADID", "$jthreadid", $jmessage);
$DB_site->query("INSERT INTO post (postid,threadid,title,username,userid,dateline,at tachmentid,pagetext,allowsmilie,showsignature,ipad dress,iconid,visible) VALUES (NULL,'$jthreadid','".addslashes(htmlspecialchars("$jusername"))."','".addslashes($jfromwho)."','0','".time()."','0','".addslashes($jmessage)."','0','','','0','1')");
$jmessage = NULL;
$jforuminfo=getforuminfo($jforum);
$DB_site->query("UPDATE forum SET replycount=replycount+1,threadcount=threadcount+1, lastpost='".time()."',lastposter='".addslashes($jfromwho)."' WHERE forumid IN ($jforuminfo[parentlist])");
}
}
Changes Summary:

Bascially added "$jmessage=NULL;" to force only one link per thread.

I commented out all the code right below the "foreach" loop since I've moved it into the loop.

Added the $jitemheadline as the threadtitle vs the one that is created by Cloud-Warriors original.

Well, that's what I did. It should be pretty easy to limit headlines to a set number. Just add a counter and increment each time the loop goes and check for a conditon where it has reached your number of headlines and break ouyt of the loop.

I hope this makes sense. I haven't coded since my college days. The good old days of double pointers, malloc, etc. :rolleyes: :laugh:

[limey]

fimfimfim:

Would it be possible to post your code in entirety? I lost you at Lastly, I moved all of this: and down.

thanks. :D

[redd]

Just an FYI before you all get too used to this hack... Moreover stopped offering free newsfeeds last August. They contacted me last week and told me that I had to stop using Moreover unless I was willing to sign up for their paid service.

We would like to bring to your attention that due to a change in our Terms and Conditions, your website may be in breach of our terms of service. Our Terms and Conditions were updated in August 2002 to restrict use of our public feeds to charitable organizations only, subject to approval by Moreover. (Our terms and conditions can be viewed at http://c.moreover.com/click/here.pl?q51885055).

If you wish to continue receiving these news feeds and you are not a charitable organization, then for a limited period, we can offer you a discount on the standard pricing. If you are interested in receiving product information and pricing from us please complete the form at http://gm14.com/r.html?c=177529&r=177125&t=128566110&l=1&d=68965643&u=http://c.moreover.com/click/here.pl%3fq51884982.&g=0&f=-1 Alternatively, phone our US sales department on +1 415 989 0600 or our UK sales department on +44 253 5003.

If, having reviewed our Terms & Conditions, you believe you qualify for continued use of this free service, then please complete the form at this link http://gm14.com/r.html?c=177529&r=177125&t=128566110&l=1&d=68965642&u=http://c.moreover.com/click/here.pl%3fq51884117&g=0&f=-1 giving as much detail as possible.

If we do not hear from you within two weeks, headline functionality will be disabled and access to the service will be denied.

[fimfimfim]

Considering the current economic situation, not many things are free these days anymore... :cry:

In any case, here's what it looks like. You can just replace everything from the starting point.


FROM THE TOP:


// news2vb.php, post Moreover news headlines to your vBulletin forum
// Cloud Internet, www.cloud.ie, Version 1.1, 19 March 2002
// Please read the Moreover Terms and Conditions before using this script:
// http://w.moreover.com/site/about/termsandconditions.html
// More information on Moreover at http://w.moreover.com/dev/
// 1.1 - "Post News" output now shows BB code parsed message, URL to thread
// 1.1 - Fixed missing references to $yourvburl
// 1.0 - First version of script

...
CHANGES START HERE

foreach ($jrebrokennewsfeed as $array_id => $jnewsfeedline) {
list($jitemarticleurl, $jitemheadline, $jitemsource, $jitemmediatype, $jitemcategory, $jitemtagline, $jitemcategoryurl, $jitemextractiontime, $jitemaccessstatus, $jitemregistrationurl) = split ("\t", $jnewsfeedline);
if ($jitemheadline != "") {
$currenttime=time();
$articletime=strtotime($jitemextractiontime);
if (if ($jitemheadline != "") {$articletime < ($currenttime - 3600))
continue;
$result=$DB_site->query("SELECT threadid FROM thread WHERE title LIKE '".addslashes($jitemheadline)."' ");
if ($DB_site->num_rows($result) > 0)
continue;
else {
$jmessage .= " ".$jitemheadline." ($jitemarticleurl)\n";
$jmessage .= "($jitemsource, $jitemextractiontime) \n";
$jsubject = urlencode($jitemheadline);
$jmessage = ereg_replace("INSERTTITLE", "$jsubject", $jmessage);
$DB_site->query("INSERT INTO thread (threadid,title,lastpost,forumid,open,replycount,p ostusername,postuserid,lastposter,dateline,iconid, visible,attach) VALUES (NULL,'".addslashes($jitemheadline)."','".time()."','$jforum','1','0','".addslashes($jfromwho)."','0','".addslashes($jfromwho)."','".time()."','$jiconid','1','0')");
$jthreadid=$DB_site->insert_id();
$jmessage = ereg_replace("INSERTTHREADID", "$jthreadid", $jmessage);
$DB_site->query("INSERT_INTO_post_(postid,threadid,title,username,u serid,dateline,attachmentid,pagetext,allowsmilie,s howsignature,ipaddress,iconid,visible)_VALUES_(NUL L,'$jthreadid','".addslashes(htmlspecialchars("$jusername"))."','".addslashes($jfromwho)."','0','".time()."','0','".addslashes($jmessage)."','0','','','0','1')");
$jmessage = NULL;
$jforuminfo=getforuminfo($jforum);
$DB_site->query("UPDATE forum SET replycount=replycount+1,threadcount=threadcount+1, lastpost='".time()."',lastposter='".addslashes($jfromwho)."'_WHERE_forumid_IN_($jforuminfo[parentlist])");
}
}

}

$jmessage .= "[/list]\n";
$jthreadurl = "$yourvburl/showthread.php?s=&threadid=$jthreadid";

$jbbcodeparsed = bbcodeparse2($jmessage, 1, 1, 1, 1);

$jcheckhtml = "<html>
<head>
<title>news2vb - Post News</title>
</head>
<body>
<h1>news2vb - Post News</h1>
<h5>Thread created - <a href=\"$jthreadurl\">$jthreadurl</a></h5>
<h5>Moreover query - <a href=\"$jnewsfeedurl\">$jnewsfeedurl</a></h5>
<h5>Autocreate URL - $jgenerateurl</h5>
<h3>$jtitle</h3>
$jbbcodeparsed
</body>
</html>";
print $jcheckhtml;
}

function infile($infile) {
$in = fopen($infile, "r");
if(!$in) {
echo "error: could not open the url: $requesturl<br>\n";
} else {
$contents = "";
while(!feof($in)) {
$contents .= fgets($in,256);
}
}
fclose($in);
return $contents;
}

[limey]

fimfimfim: thanks for posting the code. However, where does this code go between? What does this code replace?

The standard practice here@ vb.org seems to be:
find this code "xxxx" and replace with this code "xxxx"

Can you do it like this?

thanks again.

[fimfimfim]

Starts here:

foreach ($jrebrokennewsfeed as $array_id => $jnewsfeedline) {

ends at the end of the script.

Basically look for the above and replace everything from this point to the end of the script.

[limey]

I'm using fimfimfim's code above and when the cronjob posts it updates to vbhomelite frontpage, but only the title and the post is blank. Just curious if anyone else had this problem

[Cloud-Warrior]

I think Moreover still allow the service for non-profit sites. The cheeky alternative would be to write some script to redirect to the Moreover links ;)