NNTP Gateway (Usenet/Newsgroups)

This hack provides a gateway to any NNTP newsgroup such as those from Usenet and other newsgroup servers. Selected newsgroups are imported from your news server and post on the forum are sent out to the newsgroup. This hack was greatly inspired by fastforward's usenet gateway hack. This one is a complete rewrite of that hack written in PHP and works with vB 2.2.2.

Installation:
Installation is pretty simple. No modifications to the vB php files are needed and it simply consists of dropping in the files to your forums directory, running the SQL modifications, and modifying the newsgroup settings for your desired newsgroups. For the complete installation details, read the readme.txt file.

Here are some of the features:

• Real cool vbcode to text conversion!
• All posts are inserted, even if parent thread cannot be found for replies
• Guests posts also get sent to the newsgroup.
• Importing of attachments from the newsgroup
• Poll info sent to newsgroup
• Attachment link sent to newsgroup
• Supports unlimited newsgroups and servers.
• email notification of new replies, just like regular forum posts
• Can import HTML messages and convert to limited vb code.

For control panel integration, please check out GameCrash's Graphical Interface addition.

What's not in it that was in fastforward's usenet gateway hack:

• No control panel integration at the moment. (Available separately)
• Does not support separate footers
• Does not allow custom email address to be used via the users option
• No spam control or string replacement support.
• No purging or autoexpire.

There is still lots that I'd like to add to it, but this is competely functional for basic newsgroup gateway support. It is most likely not free of bugs. If you have questions, suggestions or annoyances, please post in this thread.

vb3 support
lierduh modified this to provide support for the beta versions of vb3. When the release candidates of vb3 come out, I plan to add support for vb3 and add other features to this hack. If you have suggestions for new features, let me know.

E-mail gateway support
New in this version, you can now gateway with an email list. This will fetch emails from a POP email account and send via php's mail function to the mailing list's email address. To set up the list, in the newsgroup field, enter in the email address to send outgoing posts to. In the server field, enter in the pop3 server, and in the username and password fields enter in, you guessed it, the username and password to your pop account. Note: the email gateway capabilities has not been fully tested, so it might not work with your POP server. Also, it does not work in combo with a news<->forum gateway in the same forum (you can use either kind though in differnet forums within your forums).

[Schorsch]

am I right that when:

• - I have a virtual account with 100 MB Webspace, MAX. 100 MB database
  - not allowed to run cronjobs

I can forget this useful addon ??

I'm asking since I'm really scared about that my provider is cancelling my account when I'm installing this.

cu
Schorsch

[Dmitriy]

I wonder how I can I exclude the posts in the usenet forusm from being included in the detnew and getdaily searches.

This script downloads a lot of posts from usenet and it confuses many of my members when they want to see new posts since last visit or new posts in the past 24 hours.

Can someone suggest how to hack the search.php to exclude certain forums from being counted towards the latest posts?

[Odoin]

This is something I need as well... When 500 messages are posted from the news groups a day it makes it hard for the users doing searchs.

[beheader]

Originally posted by Odoin
This might have been asked before, but there is so many posts I do not have time to search through them...

My question is... Is there a way to limit the amount of messages inported by the NNTP client? I mean like this one news group has over 5000+ messages and the client will not download them all at one shot it seems to time out before it gets done. Then I have to take the last message id and mod the forum with it, and start over but then the dates are messed up and it looks like all the messages were posted today. Any ideas?

There are only two ways I have been able to get around this was to either run a dnews server behind my firewall with a suck account from http://www.newshosting.com, or setup a newscache under linux.

Then set the limits so that only new posts are downloaded and not the 47,000 old posts.

Other than that- unless the script author makes a mod (or someone else) we are stuck with that situation.

[beheader]

Hi all,

Here's my notes on getting this mod working fulling under medium volume....

I. Installation and Config

The installation and config instructions were fine.

II. Server Choices

The mod as it stands communicates with any news server (tested 5 different servers: INN, Leafnode, Leafnode+, Dnews, Hampster, and Rumormill).

Assessment:

INN: Too industrial strength, pain to configure... don't do it (I did and now I need therapy)

Leafnode and Leafnode+: Excellent and simple. Must install from source. Ability to limit initial draw of articles from up stream server. Operates in "suck" client mode only. This is GPL software, and is currently still maintained.

Dnews: Excellent and simple. Faster than Leafnode variations. However is expensive! $95.00 for a single concurrent connection- and the prices only climb from there....

Hampster: Forget it. I'm not even going into it. But it's old, slow, and unsupported.

RumorMill: Forget it, old, Slow, and unsupported. (However if ti did work it would be a great use for an old Mac you have laying around)

My final Choice:

Leafnode.

Why? Because Leafnode compiled flawlessly, added newsgroups flawlessly, is free, is GPL, has initial suck limits in place for new group additions, has limits for message size in place (keeps some idiot who posts a binary to a text group from screwing your server and forums up, and runs on linux- which saved me a win2k license....

III. Issues with the script itself.

The mod itself works brilliantly, with the exception of the following issues:

1. If you pull a text group directly from a server with a high retention (Like 4000 articles), the download will *timeout* (Along with the php script), leaving some house keeping things un-done in regard to the import of the messages. One of these things is fairly severe, in that when the gateway script is run again, the entire newsgroup will be pulled *again*- since the message numbers are not update when the import fails.

2. There are no size limitations in place in the script proper to avoid the importing of a multipart CD image. Just today, I had to reconfigure my local newsserver to avoid a binary that was posted to comp.sys.mac.games.action- the binary should not have been there, but 680 megs of usenet posts ended up in my sql database. This is a mess and not fun.

3. It appears that the mod doesn't poll for an initial group download in the same way as a standard newsreader.

Three separate caching servers (dnews, Leafnode, and Leafnode+) all did not recognize the poll to an empty newsgroup, and subsequently download the articles. I had to subscribe to the newsgroup first in Outlook, then the messages would download normally.

While I have not had a chance to check the code for this function, I will do so in the coming days and find out if I can isolate a mod for the mod to make sure that a subscription in the control panel will be enough to trigger the newsgroup to go active.

IV. Final config- how to make it workable....

My solution consists of:

1. a suck feed from http://www.newshosting.com ($25.00 a month- 500 MB a day)

2. Internally configured Leafnode server with initial post fetch, and size limits in place.

3. CBQ (Class Based Queing) implemented on a Linux firewall to keep usenet traffic from affecting other servers on my system.

Essentially, the Leafnode server is throttled to 56k of incoming bandwidth, and the forums server connects to it via 10/100 Full Duplex Ethernet.

It was at this final config that the usenet section at ASWP.net became usable with any kind of volume. It was a big job, and very intense in the level of tweaking needed, but we have it classified as rock solid at this point.

So here's the short list of what you need to make it workable:

1. Sucking usenet server, which you have control over, or that someone is willing to alter the config for.

2. A fast semi-dedicated server box with a lot of disk space.

3. The ability to throttle your bandwidth.

Well, anyhow- thats my take on it....

[ThunderVipor]

I'm very interested in this functionality, but have no experience with MySQL (in case it isn't obvious by my subject line ). Can someone please tell me how to run the nntp.sql script?

Thanks much.

[glenvw]

This is what I am looking for for sure.

Does it actually bring visitors from the newsgroups to your site? That would be awesome.

I am looking for someone intersted in installing. Any takers?

PS I also do not know how to run the nntp.sql script.

[SeanM]

I have installed this script numerous times.. It is really quite easy, if you need some help send me a message @ webmaster@weathertalk.net and I will help you as much as possible.

SeanM

[Brian Cruz]

Is Gilby still around? I'm hoping he'll update this for vB3 when it comes out, particularly to take advantage of the new threaded view feature.

[glenvw]

oops

[fechten]

Installed and this works great for pulling down the NNTP feed, but when I try to reply to a usenet thread from the forum and then sync it up I get:

441 Posting Failed (Message-ID doesn't conform to NNTP standards)

I have html, vb code, smilies, and icons turned off for the nntp forum. Is there any other config that I should do or do posts from vb to usenet not work with this gateway?

Thanks,
Craig

[SRozhon]

Thank you so much for those pics Sanjiyan! They helped me figure out what I was doing wrong. Great idea!! Oh, and this is an awesome hack Gilby! Thanks!!

[SRozhon]

Originally posted by Gilby


It should now work with every server, and if it doesn't let me know.

I think I have it set up right now, but I keep getting an IE message that says it isn't able to connect. I am using my isp server. Is that the problem?

When I try to syncronise manually with the GUI by gateway I get this error message:
Warning: Cannot add header information - headers already sent by (output started at /home/virtual/site1/fst/var/www/html/forums/admin/adminfunctions.php:19) in /home/virtual/site1/fst/var/www/html/forums/gateway.php on line 20

Fatal error: Cannot redeclare stripslashesarray() in /home/virtual/site1/fst/var/www/html/forums/admin/global.php on line 33

Please help! This is the coolest hack I've seen yet!! :bunny:

[SRozhon]

ok, I took everything out and started over. This time I used the GUI by gamecrash. I am no longer getting the above error this time, but get this now instead.

2 group(s) gatewayed.
Could not connect to NNTP-serverLogging in to netnews.insightbb.com, group comp.graphics.apps.photoshop
Not connectedNot connectedCould not connect to NNTP-serverLogging in to netnews.insightbb.com, group comp.graphics.apps.paint-shop-pro
Not connectedNot connected

I'm not sure why it isn't connecting. I think it is set up right. Anyone have any suggestions??

[certify]

This is exaclky what I am looking for.

Great script!