NNTP Gateway (Usenet/Newsgroups)

This hack provides a gateway to any NNTP newsgroup such as those from Usenet and other newsgroup servers. Selected newsgroups are imported from your news server and post on the forum are sent out to the newsgroup. This hack was greatly inspired by fastforward's usenet gateway hack. This one is a complete rewrite of that hack written in PHP and works with vB 2.2.2.

Installation:
Installation is pretty simple. No modifications to the vB php files are needed and it simply consists of dropping in the files to your forums directory, running the SQL modifications, and modifying the newsgroup settings for your desired newsgroups. For the complete installation details, read the readme.txt file.

Here are some of the features:

• Real cool vbcode to text conversion!
• All posts are inserted, even if parent thread cannot be found for replies
• Guests posts also get sent to the newsgroup.
• Importing of attachments from the newsgroup
• Poll info sent to newsgroup
• Attachment link sent to newsgroup
• Supports unlimited newsgroups and servers.
• email notification of new replies, just like regular forum posts
• Can import HTML messages and convert to limited vb code.

For control panel integration, please check out GameCrash's Graphical Interface addition.

What's not in it that was in fastforward's usenet gateway hack:

• No control panel integration at the moment. (Available separately)
• Does not support separate footers
• Does not allow custom email address to be used via the users option
• No spam control or string replacement support.
• No purging or autoexpire.

There is still lots that I'd like to add to it, but this is competely functional for basic newsgroup gateway support. It is most likely not free of bugs. If you have questions, suggestions or annoyances, please post in this thread.

vb3 support
lierduh modified this to provide support for the beta versions of vb3. When the release candidates of vb3 come out, I plan to add support for vb3 and add other features to this hack. If you have suggestions for new features, let me know.

E-mail gateway support
New in this version, you can now gateway with an email list. This will fetch emails from a POP email account and send via php's mail function to the mailing list's email address. To set up the list, in the newsgroup field, enter in the email address to send outgoing posts to. In the server field, enter in the pop3 server, and in the username and password fields enter in, you guessed it, the username and password to your pop account. Note: the email gateway capabilities has not been fully tested, so it might not work with your POP server. Also, it does not work in combo with a news<->forum gateway in the same forum (you can use either kind though in differnet forums within your forums).

[Shaman]

I'm waiting with bated breath for a stable version of this hack... or should it be thought of as stable now?

[Gilby]

Originally posted by Shaman
I'm waiting with bated breath for a stable version of this hack... or should it be thought of as stable now?

Depends on how you define stable. right now the main issues that have risen are performance and compatibility with a variety of NNTP servers.

[Gilby]

I just uploaded a new version. Same functionality, but it should eliminate most the errors people have had. You can find that in the first post in this thread.

If you've already installed the hack, make sure you run the SQL a few posts back as that will make a huge difference for when your tables get filled with a lot of posts. In one case it brought the query speed from around 30 seconds to a fraction of a second.

If you want a gauge of how long things are taking, you can rin the script like: http://www.yourserver.com/gateway.php?showqueries=1 and it'll print out all the SQL queries as well as the start and finish time of the queries.

[GameCrash]

Well... I uploaded the new files from the new post, now I'm getting this message:

1 group(s) gatewayed.
Logging in to news.microsoft.com, group microsoft.public.german.entwickler.dotnet.csharp

Then the script exits (it's now running about 2 seconds). Nothing gets imported...

Running setlastmsg.php returns the following:

1 group(s) gatewayed.
Logging in to news.microsoft.com, group microsoft.public.german.entwickler.dotnet.csharp:
Max: newsgroup
Min: such

The lastmsg in the database stays at 0...

I'll now try the other nntp.php you posted above...

[GameCrash]

The same problem, the only difference is that there is a "12345" before the "logging in to ..."

[Erwin]

I can say that the webdiscuss.com server works with this script. FANTASTIC!!!

The script doesn't work with my ISP's newserver though. I will need to search for a free one...

[Erwin]

May I congratulate you Gilby by telling you that it works - you just need to use a public newserver that does not require a username or password.

Cool - I am hunting down one with the forums I want right now... but it works!

[GameCrash]

Now it seems to work - don't ask me why...

One thing, after import every post is set to the import date, not to the date it has really been posted to the NG. Is it possible to do that or is it a problem (timezones etc.)?

[Sanjiyan]

Gilby,

Any news on the local user replies, and why they are locking up the gateway.php ie downloads from the usenet server works, yet uploads lockup.

[Erwin]

Not true in my case - all my imported posts and threads reflect the actual time of posting, not the import date.

Wanted to say again that this is a GREAT hack! The best one I have installed, and I have installed many!

[Gilby]

Originally posted by Sanjiyan
Any news on the local user replies, and why they are locking up the gateway.php ie downloads from the usenet server works, yet uploads lockup.

Did you run the SQL update? Also try running it with the showqueries=1 in the URL.

[Gilby]

Originally posted by Erwin Loh
May I congratulate you Gilby by telling you that it works - you just need to use a public newserver that does not require a username or password.

Cool - I am hunting down one with the forums I want right now... but it works!

My two test servers that I used when developing it are "InterNetNews NNRP server INN" and one does require the username and password and it works fine. I think that it still might not work with some other nntp servers. If you have a specific server that it's not working on and you can give me access to play around with that server, I may be able to figure out what's causing the problem.

[Gilby]

Originally posted by GameCrash
1 group(s) gatewayed.
Logging in to news.microsoft.com, group microsoft.public.german.entwickler.dotnet.csharp:
Max: newsgroup
Min: such

Apparently that group does not exist, but the nntp class did not catch it as an error.

[Gilby]

Originally posted by GameCrash
One thing, after import every post is set to the import date, not to the date it has really been posted to the NG. Is it possible to do that or is it a problem (timezones etc.)?

I have the script set to import it as the date of the message if the lastmsg is set to 0. When this is set to 0, you're getting all the posts from the newsgroup. Otherwise, it'll import it and set it as the date of import, as this will make it be ordered correctly and not show up before other posts in the thread.

If you want to change how this works, you can go to line 77 and change the if-else statement that does this. So, you could comment out stuff something like:

			// if ($group[lastmsg] == 0){
				$date = strtotime($message[date]);
			//} else {
				//$date = time();
			//}

[Sanjiyan]

Originally posted by Gilby


Did you run the SQL update? Also try running it with the showqueries=1 in the URL.

I have ran the update yes, and heres the text the server gives me back, it apears to be locking up on the rec.arts.anime.misc group, which is the group where I have put my local post to on my vb server.

ALL SORRY ABOUT THE LENGTH OF THIS POST IT IS VERY LARGE:

The script stops/locksup after it checks the newsgroup just before the the newsgroup to which i have posted to, as you can see the last group checked is: rec.arts.anime.marketplace, which is the newsgroup just before the rec.arts.anime.misc group to which I have posted my reply.

---

Query: SELECT template FROM template WHERE title='options'
Time before: 0.056210041046143
Time after: 0.058135986328125

--------------------------------------------------------------------------------

Query: SELECT user.*,userfield.* FROM user LEFT JOIN userfield ON userfield.userid=user.userid WHERE user.userid='1'
Time before: 0.068742990493774
Time after: 0.070780992507935

--------------------------------------------------------------------------------

Query: INSERT INTO session (sessionhash,userid,host,useragent,lastactivity,lo cation) VALUES ('d1c7dfccf2638fdec0d342fc87b444b0','1','194.117.1 33.180','Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)','1014304767','/gateway.php?showqueries=1')
Time before: 0.08442497253418
Time after: 0.085327982902527

--------------------------------------------------------------------------------


Warning: Cannot add header information - headers already sent by (output started at /usr/home/wocsnet/public_html/global.php:120) in /usr/home/wocsnet/public_html/admin/functions.php on line 1530
Query: SELECT templatesetid,replacementsetid,userselect FROM style WHERE styleid='1'
Time before: 0.086961030960083
Time after: 0.088186979293823

--------------------------------------------------------------------------------

Query: SELECT template,title
FROM template
WHERE (title IN ('gobutton','timezone','username_loggedout','usern ame_loggedin','phpinclude','headinclude','header', 'footer','forumjumpbit','forumjump','nav_linkoff', 'nav_linkon','navbar','nav_joiner','pagenav','page nav_curpage','pagenav_firstlink','pagenav_lastlink ','pagenav_nextlink','pagenav_pagelink','pagenav_p revlink')
AND (templatesetid=-1 OR templatesetid='16'))


ORDER BY templatesetid
Time before: 0.088878035545349
Time after: 0.096379995346069

--------------------------------------------------------------------------------

Query: SELECT * FROM usergroup WHERE usergroupid=6
Time before: 0.10236001014709
Time after: 0.10398197174072

--------------------------------------------------------------------------------

Query: SELECT varname,value FROM nntp_settings
Time before: 0.11269104480743
Time after: 0.11516797542572

--------------------------------------------------------------------------------

Query: SELECT * FROM nntp_groups WHERE enabled=1
Time before: 0.11601901054382
Time after: 0.11823499202728

--------------------------------------------------------------------------------


Warning: Cannot add header information - headers already sent by (output started at /usr/home/wocsnet/public_html/global.php:120) in /usr/home/wocsnet/public_html/gateway.php on line 433
14 group(s) gatewayed.
200 Powered by Typhoon 2.0 -- http://www.highwinds-software.com/ (Typhoon v2.0.0)
381 More Authentication Required
502 Authentication Failed
211 99 3216690 3216800 rec.arts.anime
Logging in to news-text.blueyonder.co.uk, group rec.arts.anime:
Query: SELECT post.postid,post.username,post.userid,post.datelin e,post.attachmentid,post.pagetext,post.showsignatu re,post.ipaddress,thread.pollid,thread.title,post. threadid,thread.forumid FROM thread,post WHERE post.isusenetpost=0 AND thread.threadid=post.threadid AND post.dateline=thread.dateline AND post.userid=thread.postuserid AND (thread.forumid=59)
Time before: 0.16228294372559
Time after: 0.17449402809143

--------------------------------------------------------------------------------

Query: SELECT post.postid,post.username,post.userid,post.datelin e,post.attachmentid,post.pagetext,post.showsignatu re,post.ipaddress,thread.title,post.threadid,threa d.forumid,thread.msgid,thread.prefix FROM thread,post WHERE post.isusenetpost=0 AND thread.threadid=post.threadid AND (thread.forumid=59)
Time before: 0.17489802837372
Time after: 0.18513000011444

--------------------------------------------------------------------------------

200 Powered by Typhoon 2.0 -- http://www.highwinds-software.com/ (Typhoon v2.0.0)
381 More Authentication Required
502 Authentication Failed
211 117 3216740 3216874 rec.arts.anime.creative
Logging in to news-text.blueyonder.co.uk, group rec.arts.anime.creative:
Query: SELECT post.postid,post.username,post.userid,post.datelin e,post.attachmentid,post.pagetext,post.showsignatu re,post.ipaddress,thread.pollid,thread.title,post. threadid,thread.forumid FROM thread,post WHERE post.isusenetpost=0 AND thread.threadid=post.threadid AND post.dateline=thread.dateline AND post.userid=thread.postuserid AND (thread.forumid=60)
Time before: 0.22879302501678
Time after: 0.29553496837616

--------------------------------------------------------------------------------

Query: SELECT post.postid,post.username,post.userid,post.datelin e,post.attachmentid,post.pagetext,post.showsignatu re,post.ipaddress,thread.title,post.threadid,threa d.forumid,thread.msgid,thread.prefix FROM thread,post WHERE post.isusenetpost=0 AND thread.threadid=post.threadid AND (thread.forumid=60)
Time before: 0.29594302177429
Time after: 0.35159695148468

--------------------------------------------------------------------------------

200 Powered by Typhoon 2.0 -- http://www.highwinds-software.com/ (Typhoon v2.0.0)
381 More Authentication Required
502 Authentication Failed
211 293 3235896 3236281 rec.arts.anime.fandom
Logging in to news-text.blueyonder.co.uk, group rec.arts.anime.fandom:
Query: SELECT post.postid,post.username,post.userid,post.datelin e,post.attachmentid,post.pagetext,post.showsignatu re,post.ipaddress,thread.pollid,thread.title,post. threadid,thread.forumid FROM thread,post WHERE post.isusenetpost=0 AND thread.threadid=post.threadid AND post.dateline=thread.dateline AND post.userid=thread.postuserid AND (thread.forumid=67)
Time before: 0.40430796146393
Time after: 0.43287003040314

--------------------------------------------------------------------------------

Query: SELECT post.postid,post.username,post.userid,post.datelin e,post.attachmentid,post.pagetext,post.showsignatu re,post.ipaddress,thread.title,post.threadid,threa d.forumid,thread.msgid,thread.prefix FROM thread,post WHERE post.isusenetpost=0 AND thread.threadid=post.threadid AND (thread.forumid=67)
Time before: 0.43329298496246
Time after: 0.45920097827911

--------------------------------------------------------------------------------

200 Powered by Typhoon 2.0 -- http://www.highwinds-software.com/ (Typhoon v2.0.0)
381 More Authentication Required
502 Authentication Failed
211 15 3215061 3215129 rec.arts.anime.games
Logging in to news-text.blueyonder.co.uk, group rec.arts.anime.games:
Query: SELECT post.postid,post.username,post.userid,post.datelin e,post.attachmentid,post.pagetext,post.showsignatu re,post.ipaddress,thread.pollid,thread.title,post. threadid,thread.forumid FROM thread,post WHERE post.isusenetpost=0 AND thread.threadid=post.threadid AND post.dateline=thread.dateline AND post.userid=thread.postuserid AND (thread.forumid=68)
Time before: 0.48813498020172
Time after: 0.49104905128479

--------------------------------------------------------------------------------

Query: SELECT post.postid,post.username,post.userid,post.datelin e,post.attachmentid,post.pagetext,post.showsignatu re,post.ipaddress,thread.title,post.threadid,threa d.forumid,thread.msgid,thread.prefix FROM thread,post WHERE post.isusenetpost=0 AND thread.threadid=post.threadid AND (thread.forumid=68)
Time before: 0.4914870262146
Time after: 0.49426400661469

--------------------------------------------------------------------------------

200 Powered by Typhoon 2.0 -- http://www.highwinds-software.com/ (Typhoon v2.0.0)
381 More Authentication Required
502 Authentication Failed
211 0 2 1 rec.arts.anime.info
Logging in to news-text.blueyonder.co.uk, group rec.arts.anime.info:
Query: SELECT post.postid,post.username,post.userid,post.datelin e,post.attachmentid,post.pagetext,post.showsignatu re,post.ipaddress,thread.pollid,thread.title,post. threadid,thread.forumid FROM thread,post WHERE post.isusenetpost=0 AND thread.threadid=post.threadid AND post.dateline=thread.dateline AND post.userid=thread.postuserid AND (thread.forumid=69)
Time before: 0.52041101455688
Time after: 0.5222350358963

--------------------------------------------------------------------------------

Query: SELECT post.postid,post.username,post.userid,post.datelin e,post.attachmentid,post.pagetext,post.showsignatu re,post.ipaddress,thread.title,post.threadid,threa d.forumid,thread.msgid,thread.prefix FROM thread,post WHERE post.isusenetpost=0 AND thread.threadid=post.threadid AND (thread.forumid=69)
Time before: 0.52265894412994
Time after: 0.52438294887543

--------------------------------------------------------------------------------

200 Powered by Typhoon 2.0 -- http://www.highwinds-software.com/ (Typhoon v2.0.0)
381 More Authentication Required
502 Authentication Failed
211 456 3258414 3259022 rec.arts.anime.marketplace
Logging in to news-text.blueyonder.co.uk, group rec.arts.anime.marketplace:
Query: SELECT post.postid,post.username,post.userid,post.datelin e,post.attachmentid,post.pagetext,post.showsignatu re,post.ipaddress,thread.pollid,thread.title,post. threadid,thread.forumid FROM thread,post WHERE post.isusenetpost=0 AND thread.threadid=post.threadid AND post.dateline=thread.dateline AND post.userid=thread.postuserid AND (thread.forumid=70)