NNTP Gateway (Usenet/Newsgroups)

This hack provides a gateway to any NNTP newsgroup such as those from Usenet and other newsgroup servers. Selected newsgroups are imported from your news server and post on the forum are sent out to the newsgroup. This hack was greatly inspired by fastforward's usenet gateway hack. This one is a complete rewrite of that hack written in PHP and works with vB 2.2.2.

Installation:
Installation is pretty simple. No modifications to the vB php files are needed and it simply consists of dropping in the files to your forums directory, running the SQL modifications, and modifying the newsgroup settings for your desired newsgroups. For the complete installation details, read the readme.txt file.

Here are some of the features:

• Real cool vbcode to text conversion!
• All posts are inserted, even if parent thread cannot be found for replies
• Guests posts also get sent to the newsgroup.
• Importing of attachments from the newsgroup
• Poll info sent to newsgroup
• Attachment link sent to newsgroup
• Supports unlimited newsgroups and servers.
• email notification of new replies, just like regular forum posts
• Can import HTML messages and convert to limited vb code.

For control panel integration, please check out GameCrash's Graphical Interface addition.

What's not in it that was in fastforward's usenet gateway hack:

• No control panel integration at the moment. (Available separately)
• Does not support separate footers
• Does not allow custom email address to be used via the users option
• No spam control or string replacement support.
• No purging or autoexpire.

There is still lots that I'd like to add to it, but this is competely functional for basic newsgroup gateway support. It is most likely not free of bugs. If you have questions, suggestions or annoyances, please post in this thread.

vb3 support
lierduh modified this to provide support for the beta versions of vb3. When the release candidates of vb3 come out, I plan to add support for vb3 and add other features to this hack. If you have suggestions for new features, let me know.

E-mail gateway support
New in this version, you can now gateway with an email list. This will fetch emails from a POP email account and send via php's mail function to the mailing list's email address. To set up the list, in the newsgroup field, enter in the email address to send outgoing posts to. In the server field, enter in the pop3 server, and in the username and password fields enter in, you guessed it, the username and password to your pop account. Note: the email gateway capabilities has not been fully tested, so it might not work with your POP server. Also, it does not work in combo with a news<->forum gateway in the same forum (you can use either kind though in differnet forums within your forums).

[Gilby]

Originally posted by CJi
No matter *what* I do, all I can get out of gateway.php is this:

1 group(s) gatewayed.
Logging in to news.usenet.com, group alt.sports.airsoft

nothing else.

Try the attached nntp.php file in place of the one you are using. this will output the responses from the nntp server.

[Gilby]

Originally posted by Dontom
Gilby... Thanks for the great hack!!

I would like to exclude Usenetposts which contain a x-archive = no in their header - has anyone already done this? or is there a hint at which part of the script i can sort those posts out?
Thank you!
Tom

Look at the statement that checks if it is a forum post that is being read from teh newsgroup (if ($message[useragent] != $settings[useragent]). Add an exclusion for that header too. You may need to modify something else so that the header shows up in the message array.

[Gilby]

Originally posted by akaler

Any idea how to filter out the bad seeds? The hacky idea I have above would work if only imported posts, not just threads, could somehow be attached to a username. I know vb.org accomplished this somehow. They grabbed the hacks forums from vbulletin.com, then when people registered here, it would attach their user name to the name they used on their old messages, and whala... full vb user based functions could be applied to old messages.

A simple SQL query can be used periodically to delete them. Use something like "DELETE FROM post WHERE username='name of bad seed'"

You'll also need to run DELETE FROM thread WHERE postusername='name of bad seed'"

Or you can see my last post and add another thing to the if statement.

[CJi]

ok, thanks for that Gilby. I've got a bit more information. It logs in ok, but then does nothing? I tried it with uk.test last night and got some action, but there are posts in alt.sport.airsoft (not many granted), but still some. I'd have thought it would at least import those few? Here's what I got:

1 group(s) gatewayed.
200 Welcome to Usenet.com, 5.5g6, S8
381 PASS required
281 Ok
211 6 621 626 alt.sport.airsoft selected
Logging in to news.usenet.com, group alt.sport.airsoft

[akaler]

Originally posted by Gilby
A simple SQL query can be used periodically to delete them. Use something like "DELETE FROM post WHERE username='name of bad seed'"

You'll also need to run DELETE FROM thread WHERE postusername='name of bad seed'"

Or you can see my last post and add another thing to the if statement.

I think that vB deletes threads by username, even usersnames used in usenet posts. But the SQL query you gave me to delete posts by username... wow, that's useful!

So is the filter heading instructions. Great info, thanks!!

BTW... I've found two possible issues...

The first is posting to Supernews. I couldn't get it to post WITHOUT removing "Message-ID: <$msgid>\r\n" from the $response string. Supernews always ACCEPTED the messages just fine (240), but they just weren't showing up. I guess that either they have a beef with the way you generate the msgid OR they prefer to generate their own, and their server gets "insulted" by the fact you're offering your own. I don't know.

This didn't happen using Giganews, incidentally. Everything posted there just fine w/o modification. I only moved to Supernews a couple days ago, and still have my Giga account 'till the end of the month in fact. I believe it always worked well with Easynews (they have a $1 trial I checked out).

Side note to others reading -- Those services I just mentioned are probably the "big 3" of services offering quality feeds with high rentention, in case anybody is interested. If you want something cheaper but still good ($5mo), check out airnews.net. SUPER cheap (free) but still quality is readfreenews.net.

Reading access is free and needs no password. To get posting access to readfreenews, net, you have to email drechsau+usenet@yuck.net (Mike Horwath), with real name, real email isp email address, username wanted, password wanted.


Alright... next possible bug...

Sometimes when pulling groups that have an a large number of messages.. generally 25-80 thousand... the httpd process (with php as a module) begins swelling in memory access VERY VERY rapidly. Say 2mb growth every few seconds. After 1000 messages, I might have a 400mb process! It will then proceed to fill up the whole of available memory AND swap. But it'll still work. After the first time this happened, I've been killing these processes gone wild, though.

Sometimes this doesn't happen. Sometimes I can tell it to grab 40k messages or whatever... and see perfectly ordinary memory usage... a growth of just a few k every few seconds. Normally, it ends up being something like a 130mb or so after tends of thousands of message havebeen gotten, not just a few hundred.

This tends to happen more AFTER a reboot or a successul restart of Apache. What this seems to indicate to me is that some sort of memory routine isn't being initialized properly... but after php and apache have been used for awhile... everything that could be initialized in apache has been... That's just my theory, I could be wrong.

No matter what it is, though, this might be more of a php issue (i'm using the latest non-beta self-compiled version up on the php.net site... 4.1.2) than a problem with your script. Since it's memory related, I even thought it might be kernel related, but I just updated quite a few revisions and it still happens. I even recompiled php with different options. Same problem. Guess I'll have to wait for the stable release of 4.2 to see if it still happens.

Alrighty... thanks a bunch for your help and continued attention, Gilby! This is a great hack. You should really be compensated for this. Consider having some sort of donation system that parallels vbportals. I say vb portals, because the combination of everything they do to get donations is very successful. Offering betas only if you've donated and the other things they do really does seem to get them more donations than normal.

[CJi]

Is it at all possible that, when a local (vB) reply to a newsgroup is deleted, it gets cancelled off the newsgroup as well?

Just a thought

[The Realist]

Installed ok but Im getting the following error:

<br>
<b>Warning</b>: Failed opening 'class.POP3.php3' for inclusion (include_path='.:/php/includes:/usr/share/php') in <b>/home/virtual/site1/fst/var/www/html/forum/gateway.php</b> on line <b>25</b><br>
<br>
<b>Warning</b>: Cannot add header information - headers already sent by (output started at /home/virtual/site1/fst/var/www/html/forum/gateway.php:25) in <b>/home/virtual/site1/fst/var/www/html/forum/gateway.php</b> on line <b>483</b><br>
0 group(s) gatewayed.

Any help?

Brian

[The Realist]

Uploaded class.POP3.php3 to the forums dir and now no errors. But all Im getting is this:

0 group(s) gatewayed.

and no posts from newsgroup in the section?

[Gilby]

Originally posted by akaler
The first is posting to Supernews. I couldn't get it to post WITHOUT removing "Message-ID: <$msgid>\r\n" from the $response string. Supernews always ACCEPTED the messages just fine (240), but they just weren't showing up. I guess that either they have a beef with the way you generate the msgid OR they prefer to generate their own, and their server gets "insulted" by the fact you're offering your own. I don't know.

Most news clients generate their own message ids so I'd guess that there is something with the format that the message id is in. I'm not aware on if there is a standard format for the message id, just that it needs to be unique, but I'll look into it to see if there is one that I was not aware of. It could also be that the supernews server doesn't accept a message id that has the host part of it different from where it's being sent from, so you may want to look at the posts that are made to it and see what the message id looks like. Then use that host for the setting in the nntp gateway.

Originally posted by akaler
Sometimes when pulling groups that have an a large number of messages.. generally 25-80 thousand... the httpd process (with php as a module) begins swelling in memory access VERY VERY rapidly.

Hmmm... I hope it's a problem with something other than my code. However, I'll take a look at this when I get a chance and maybe try out some large imports to see if I can duplicate this.

Meanwhile, you can set a limit of how many posts you want per pull per newsgroup by putting the following code before the line that contains (for ($current = $lastmsg+1; $current <= $max; $current++)), about line 90:

	if ($max - $current > 1000) {
		$max = $current + 1000;
	}

If it's a problem with my code, then php will free the memory when it's done executing the script (it may also do this is the problem is elsewhere).

Originally posted by akaler

Alrighty... thanks a bunch for your help and continued attention, Gilby! This is a great hack. You should really be compensated for this. Consider having some sort of donation system that parallels vbportals. I say vb portals, because the combination of everything they do to get donations is very successful. Offering betas only if you've donated and the other things they do really does seem to get them more donations than normal.

I've been thinking of possibly calling it "donationware" which simply means if it's useful for you and you can afford to contribute some cash, then please donate. So, if you want to donate, you can use this PayPal link: https://www.paypal.com/xclick/busine...r=nntp-gateway

[Gilby]

Originally posted by The Realist
Uploaded class.POP3.php3 to the forums dir and now no errors. But all Im getting is this:

0 group(s) gatewayed.

and no posts from newsgroup in the section?

Did you set up a newsgroup to import? In the nntp_groups table, you need to add a row that has the server, newsgroup, username, password, etc for the newsgroup you want to gateway in. If you don't know how to add a row in the mysql table, you may want to install the GUI that was made for this hack: https://vborg.vbsupport.ru/showthrea...threadid=36475