NNTP Gateway for Usenet ( Newsgroups ), Mailing Lists

The latest version: V2.3.2 (31 Oct 2004)

What is it?

This hack provides a gateway from vBulletin forums to the USENET. It allows forum members to read newsgroup messages as well as to send posts to the newsgroups. In a nutshell, it turns vBulletin forums into a basic function news reader.

For an added bonus, this package also supports mailing list archive and forum to mailing list gateway. The software retrieves messages from a pop3 account and import them into the forums threaded. It also allows a forums user to send messages to the mailing list by the forum posting interface.

Why?

The gateway will be appreciated by forum members who do not know how to set up a news reader, or people who do not have port 119 open in their network. A forum interface to read and post news are often considered to be better than a news reader.

The gateway adds more contents to your forums and allows the members to interact with wider audience. It is common to see new users signing up to use this feature.

History

This hack is basically a hacked vB2 NNTP gateway written by Gilby to work with vB3. The original hack can be found here:

https://vborg.vbsupport.ru/showthrea...7&page=1&pp=15


Features

Some of the features and improvement include:

• Handles both newsgroup posts and mailing lists.
• Handles MIME encoded messages and attachment.
• Handles UUEncoded messages.
• Two Admin Control Panels for all settings.
• Automated installation.
• Correct thread implementation. All imported messages and posts sent to newsgroups will be threaded correctly according to References and Message-ID headers.
• Optional setting to thread by subject for mailing lists that strip references header.
• Supports TABLE_PREFIX.
• Imported messages will be searchable just like normal forum posts.
• Similar threads are built when the messages are imported if the system setting is on.
• Multiple attachments importing.
• Save attachment to database or file according to vB setting.
• Thumbnail created while importing image files.
• Multiple attachment post to newsgroup is also supported.
• Selectable X-No-Archive header honouring system.
• Light on system resources.
• Global Killfile to filter out unwanted messages. Killfiles can have OR and AND effectiveness.
• Strip footer by setting for incoming messages. Strips out repeated message footers.
• Built options to disable signatures and footers for out going messages.

Future development

• AdminCP interfaces. (completed)
• Installation/upgrade script. (completed, coded into AdminCP)
• Traditional "> " lead posting interface.

What do you need and do not need?

You do not need to edit vB php files if you use two bookmarks
You need to edit one vB php file to insert two lines of code if you want direct click links within AdminCP.
No need to run SQL.

Installation

Basically the AdminCP interfaces take care of installation as well as upgrade in the background. Back up your database first. The scripts can modify your database without alerting you!

Please follow the instructions in the readme.txt file for more details.

Support

Please post your support questions to this thread. Please check the FAQ in the third post within this thread first.

================== UPDATES ======================
Jul 31 2004 (release version 2.2.0)

• Provided option of not importing attachments
• Provided option of pause between messages
• Provided option of not sending messages to USENET
• Provided mechanism to skip importing a message if it was crashed on it
• Provided "Default Footer" option
• Provided strip_footer function for mailing lists
  (written originally by KevinM)
• Provided Killfile functions with Header/Body/AND/OR settings
• Provided ability to thread messages by subject for mailing lists
• Provided ability to use a single POP3 account for different mailing lists
  (use prefix)
• Merged KevinM's attachment routine, save thumbnail to hard drive
  as vB3.0.3
• Official support to have the script run by Scheduled Tasks
• Many bug fixes

Please note, when you manually run the script, you must turn on the debug mode. Otherwise nothing will be displayed on the screen.

To run as debug mode: http://forums.yourdomain.com/gateway.php?debug=1

25 Oct 2004

• Provided the option of sending posts using the real email address
• Provided the option of sending posts using a fixed email address for
  each group.
• Posts are now held back if they are in the moderating queue
• Scheduled task log is now working
• Provided the option of sending the debug messages to the vB3 cron log
• Bug fixes

31 Oct 2004

Fixed bugs running the gateway by the vB3 Scheduled Tasks

1 March 2004

Uploaded the new zip file. It contains one or two bug fixes.

[Imhotep]

AWESOME!!! Thanks Kevin! Your hard work is appreciated. This will go along way towards making this hack a little bit more secure and BB friendly.

[Imhotep]

Hmmm.... it seems I am now getting an error. I'm not sure if this is related to the new attachment perms or not.

Content-type: text/html
X-Powered-By: PHP/4.3.4

<br />
<b>Parse error</b>:  parse error in
<b>/home/cghaus2/public_html/forums/gateway.php</b> on line <b>1079</b><br />

I also upgraded to the new version this morning.

[KevinM]

Originally Posted by Imhotep

Hmmm.... it seems I am now getting an error. I'm not sure if this is related to the new attachment perms or not.

Maybe I made a mistake writing the instructions. I have attached a version of gateway.php which is working for me with the attachment permissions.

Try this.

[Princeton]

thank you Kevin ... no problems on my end

Imhotep,
ReCheck your code ... you probably made a mistake copying / pasting

[Imhotep]

Originally Posted by KevinM

Maybe I made a mistake writing the instructions. I have attached a version of gateway.php which is working for me with the attachment permissions.

Try this.

Kevin, thanks, this file fixed the problem. I am usually very careful about how I copy/paste, so I am not sure if I made a mistake or not. I suppose if princeton got it working, then I probably fulbbed somewhere along the line.

thanks again

[BustaCap]

In the NNTP Gateway setting, under the "Email address domain" option, I'm not quite sure what to put there. What do you mean by

To avoid spams attempting at your server. Please do not use a resolvable domain.

Do I need to change the default value?

Also, for the "Mailing list E-mail" setting, do I put the email address of the mailing list (eg, coolgroups@yahoogroups.com) or is it the email address that will be sending messages to the mailling list (eg, me@mydomain.com). The yahoo mailing list that I want to archive only accepts emails from verified email addressess. Is that going to be a problem?

Finally, two suggestions:

1. For mailing lists, all emails will have a common word or phrase in the subject. For example, the subject of all messages for the "New DVD Releases" mailing list would look something like:

[NewDVD] Spider Man deluxe edition
RE: [NewDVD] Spider Man deluxe edition
etc...

It would be great if you could add a setting in the gateway setup that would allow the user to specify the subject phrase (eg, [NewDVD]) and then strip that from the subject text so that it doensn't show up in the title of every post. Also, even more importantly, have a check box that would allow us to only import message that has the subject phrase. This would allow us to filter out messages that are not actually mailing list messages.

2. Every mailing list message that I get has the same text in the footer. It would be great if there was a setting somehwere where I could specify this footer text (its over 13 lines!) and have it stripped before importing the message.

Thanks, this hack is really useful for me!

[KevinM]

I've been thinking about attempting a 'killfile' feature (a common newsreader feature) which would allow the administrator to set some parameters and stop the downloading of certain posts into the forum.

I propose it works like this:

In the admin control panel, you enter the phrases (surrounded by {} ) which you would not want downloaded (similar to the censorship feature).

As each post is processed, it checks the headers, author and title of a message to see if it matches with the phrases, and if it does, discard it.

Possible uses:
You could stop a specific troll's posts from reaching your boards by putting the posters name into the killfile (although any replies by other usenet users will get through).

You could block messages originating from a specific server (for trolls that keep changing their posting name), although you risk blocking anyone else using the same server (small risk).

You could set up common 'spam' message titles to block.

Downsides:
Performance. Putting dozens of words or phrases into the killfile will slow down the processing of messages while the gateway script is being run.

Pureists. Some people using the newsgroups may consider this a form of moderation in an unmoderated group.

I'd really appreciate others users of this scripts thoughts.

Is this worth pursuing and is it the right approach?

[BustaCap]

Sounds good. Would be great if it worked the other way as well; only posts with a certain word(s) in the title would be posted to the board (eg, see my request #1 in my above post).

[KevinM]

Originally Posted by BustaCap

Sounds good. Would be great if it worked the other way as well; only posts with a certain word(s) in the title would be posted to the board (eg, see my request #1 in my above post).

This could work for mailing lists, but not newsgroups. Also, each mailing list would require a different keyword to be matched in the title.

I agree about the footer. Again you would need to accomodate multiple different footers (for each mailing list, or for groups like yahoogroups which regularly change their footers from day to day). Any risk to possible violations of the terms and conditions from the list operators?

[BustaCap]

Yes, I was referring to mailings lists. Ideally, I would like to see a form for every mailing list, which would be required for any "catch phrases".

At the moment none of the posts that I get for my mailings lists are being threaded; they are all only one post deep and each reply shows up as a seperate post. It looks the "RE:" portion of the subject title might be causing problems since it can show up in several different spots in the title. For example, for the original thread, [NewDVD] Spider Man deluxe edition, the replies show up as :

RE: [NewDVD] Spider Man deluxe edition
[NewDVD] RE: Spider Man deluxe edition
RE: [NewDVD] RE: Spider Man deluxe edition

I'm guessing it's just how the various mail programs that people use handle inserting "RE". Looking at several hundred messages, the best way to determine which reply goes with which message would be to look at the title starting from the end of the message and working backwards. You don't usually see the [xxx] or the RE: part at the end of the subject. A quick browse through about 300 messages showed that if I used the last three or four words of the subject line it would have accurately put the replies with the correct original thread.