[RELEASE] Usenet Gateway for vB 2.01

Description:

This hack provides a gateway to selected usenet groups. The entire usenet group is mirrored locally. Posts submitted locally to mirrored newsgroups will also be sent out to usenet. A single perl script is executed via cron that collects news and posts any outgoing articles. Incoming messages are parsed against customizable spam filters and threaded correctly before being placed in your forums. By default, no personal information is sent to usenet other than the posters username and whatever you have globally configured as a footer and organization. The option does exist for a user to include a custom email to use for usenet posts. After initial installation (which involves creating some new tables, adding a few columns to existing vB tables, and entering the database connection information to the news script) configuration is completed via the vB control panel.

• See it in action at dBforums.com (usenet forums are at the bottom)
• Download latest version for vB 2.2.4

Current version is 20020323 for vB2.xx

Current feature list:

• correct threading of outgoing posts on usenet
• full control panel integration
• email notification to usenet replies
• emoticon translation into vb icons
• vb code removal or conversion prior to posting to usenet
• hyperlinked urls in messages
• color coded, italicized & indented quotes
• vB style quote to usenet style quote conversions for outgoing posts
• logging of outgoing posts
• support for multiple news servers
• support for seperate footers per forum in outgoing posts
• multi-language (selectable) handling of quoted MIME printable headers (for all those funny foreign characters)
• handling of mailing archive groups
• flexible spam control and replacement variable options for incoming and outgoing messages
• configurable auto-expire option
• option for users to show email address in their outgoing usenet posts (may be different than normal one in profile
• option to enable/disable user signatures on outgoing posts.
• fully compatible with vB moderation functions

Still to be done:

• canceling of messages after they have been sent to usenet via control message in accordance with RFC1036.
• binary attachment support
• email to PM gateway

Screeshots:
Main Options
More Options
Configuration of Newsgroups
Add New Newsgroups
Spam Control
What an imported usenet article looks like in vB

The latest version of this hack will always be in this first post of the thread.

Before asking questions in this thread, please read all the posts in this thread. If your question goes unanswered, it will probably be because the question has already been answered countless times in this thread.

This hack was created for use on my forums and will only be supported as time permits.


Links to required Perl Modules

• News::NNTPClient
• Net::SMTP
• MIME::WordDecoder
• Text::AutoFormat
• Mail::Address
• URI::Find
• DBI
• DBD::MySQL
• Date::Parse
• HTML::Entities

[fastforward]

Release 20010729 fixes two problems with newnews.pl.

• Fixed SQL error when post is made with signature set to not be shown.
• Fixed posting time for local posts made in usenet forum. The nntpdateline was being shown as 1969.

To upgrade, you just need to replace your existing newnews.pl with the one in this package.


The latest version of this hack will always be in this first post of the thread.

[James Cridland]

Installing this on an ActivePerl build for Windows 2000.

So far, I've managed to get further installing the modules. ActivePerl uses PPM modules. To install, type "PPM" into a command box, then type "install [package name]".

Which is where I've managed pretty well, until the last three. Package names differ slightly, but this is what I've installed so far, with their CPAN equivalent names...


cpan> install News::NNTPClient
News-NNTPClient

cpan> install Net::SMTP
NetServer-SMTP

cpan> install DBD::mysql::Install
DBD-Mysql

cpan> install Text::Autoformat
Text-Autoformat

cpan> install Date::Parse
Date-Manip

cpan> install MIME::Tools
MIME-tools

cpan> install URI::Find
URI-Find

cpan> install MIME::WordDecoder
?

cpan> install HTML::Entities
?

cpan> install Mail::Address
?

...and as you can see, I'm stuck with the last three. I'm still persevering, and I hope this is useful to some who are trying to install it on my kind of setup (ActivePerl, Apache, Win2000).


Until then, I've found a script which offers both read and write functionality just through PHP - check out http://forums.mediauk.com/usenet/ for a demonstration and a link to the source. However, it doesn't use mySQL or indeed any database structure (probably quite easy to patch for a programmer who knows what they're doing, mind.)

It would make more sense, to this user at least, to have a fully native PHP solution rather than having to use Perl: since, as I understand it, all the necessary code's already in PHP.

[webhost]

I am currently using this version 20010528,

What do I need to do to upgrade just upload the newnews.pl or more?

[fastforward]

Originally posted by webhost
I am currently using this version 20010528,

What do I need to do to upgrade just upload the newnews.pl or more?

There's other changes since that version. There's also aditional modules that are needed. The best bet is to download the interim packages upgrade in order.
usenet_gateway_v20010714__201.tar.gz
usenet_gateway_v20010721__201.tar.gz
Then finally, do the latest upgrade from this thread.

[fastforward]

Originally posted by James Cridland
cpan> install MIME::WordDecoder
?
cpan> install HTML::Entities
?
cpan> install Mail::Address
?

...and as you can see, I'm stuck with the last three. I'm still persevering, and I hope this is useful to some who are trying to install it on my kind of setup (ActivePerl, Apache, Win2000).

Activestate provide ports for all of these
WordDecoder is part of the MIME-Tools package.
HTML::Entities is part of the HTML-Parser package
Mail::Address is part of the MailTools package

I recently changed some of the modules that were being used to ensure that there were ActiveState modules available.

On win32 systems, you will have to comment out the bit that checks for a previous running version near the top of newnews.pl.

It would make more sense, to this user at least, to have a fully native PHP solution rather than having to use Perl: since, as I understand it, all the necessary code's already in PHP.

There are no native NNTP functions built into php. There is IMAP support (assuming you compiled it with the appropriate options), but that doesn't provide you with full newsreader capabilities. That aside, the reading of the news is actually the easy bit. The main guts of the script deals with news synchonisation & parsing and database access and vB integration. There is no doubt you could write it in PHP, but you wouldn't have the Perl modules to help you. You would also have to install the binary version of php to allow periodic execution of the script.

[James Cridland]

I'm nearly there!
I've made some additions to the installation for other W2K/ActivePerl users, which (once everything works) I'll post here.

My guess at "Date-Manip" in PPM was the wrong package to install. Turns out the right one is "TimeDate", and not Date-Manip.

[fastforward]

Originally posted by James Cridland
I'm nearly there!
I've made some additions to the installation for other W2K/ActivePerl users, which (once everything works) I'll post here.

My guess at "Date-Manip" in PPM was the wrong package to install. Turns out the right one is "TimeDate", and not Date-Manip.

Glad you're getting there. If you don't mind I'll include your ActiveState instructions with the package once you've got it all going. I'll also see if there's any other way of checking for an existing instance without using grep that will work on both Unix and Win32. In the meantime, you'll just have to comment it out.

[James Cridland]

Here's the additional stuff, which you can include in any future releases. The numbers at the beginning are the original installation notes. I hope it's useful.

Initial thoughts are, certainly when using news.btinternet.com (my local ISP), that newnews.pl is very slow in grabbing new messages - typically up to 90 seconds per message. For this reason, I'm currently setting it to run once a day. Is this about right?

It strikes me that running a local news server would speed this process up; is there a suitable freeware news server available somewhere?

This is a brilliant addition; now in use (though not extensively) at http://forums.mediauk.com/

1. [Installing the modules]
WIN32/ActivePerl: To install these, type "PPM" into a command box. Then, type:
install News-NNTPClient
install NetServer-SMTP
install DBD-Mysql
install Text-Autoformat
install TimeDate
install MIME-tools
install URI-Find

3. [Running the sql scripts]
WIN32: To run these using mysql: type "mysql" at the command prompt. Then, type:
connect vbfor [where vbfor is the name of your database]
source vB_sql_structure.sql
source vB_sql_data.sql

8. [Editing newnews.pl]
WIN32: Comment out line 53 - line 67.

Running:
WIN32: Set up a shortcut (either on your desktop or in your start menu) to point to: "perl c:\path\newnews.pl"
Then, use Scheduler (Start|Programs|Accessories|System Tools|Scheduled Tasks) to run this program at specified times.

[fastforward]

Originally posted by James Cridland
Initial thoughts are, certainly when using news.btinternet.com (my local ISP), that newnews.pl is very slow in grabbing new messages - typically up to 90 seconds per message. For this reason, I'm currently setting it to run once a day. Is this about right?

90 seconds is most definately NOT about right. It shouldn't even take that long for a full batch of 200 or 300. try putting some debugging lines in the code to see exactly where it goes wrong.

[James Cridland]

Well, I might need your help here...

I'll have a play tomorrow.