[RELEASE] Usenet Gateway for vB 2.01

Description:

This hack provides a gateway to selected usenet groups. The entire usenet group is mirrored locally. Posts submitted locally to mirrored newsgroups will also be sent out to usenet. A single perl script is executed via cron that collects news and posts any outgoing articles. Incoming messages are parsed against customizable spam filters and threaded correctly before being placed in your forums. By default, no personal information is sent to usenet other than the posters username and whatever you have globally configured as a footer and organization. The option does exist for a user to include a custom email to use for usenet posts. After initial installation (which involves creating some new tables, adding a few columns to existing vB tables, and entering the database connection information to the news script) configuration is completed via the vB control panel.

• See it in action at dBforums.com (usenet forums are at the bottom)
• Download latest version for vB 2.2.4

Current version is 20020323 for vB2.xx

Current feature list:

• correct threading of outgoing posts on usenet
• full control panel integration
• email notification to usenet replies
• emoticon translation into vb icons
• vb code removal or conversion prior to posting to usenet
• hyperlinked urls in messages
• color coded, italicized & indented quotes
• vB style quote to usenet style quote conversions for outgoing posts
• logging of outgoing posts
• support for multiple news servers
• support for seperate footers per forum in outgoing posts
• multi-language (selectable) handling of quoted MIME printable headers (for all those funny foreign characters)
• handling of mailing archive groups
• flexible spam control and replacement variable options for incoming and outgoing messages
• configurable auto-expire option
• option for users to show email address in their outgoing usenet posts (may be different than normal one in profile
• option to enable/disable user signatures on outgoing posts.
• fully compatible with vB moderation functions

Still to be done:

• canceling of messages after they have been sent to usenet via control message in accordance with RFC1036.
• binary attachment support
• email to PM gateway

Screeshots:
Main Options
More Options
Configuration of Newsgroups
Add New Newsgroups
Spam Control
What an imported usenet article looks like in vB

The latest version of this hack will always be in this first post of the thread.

Before asking questions in this thread, please read all the posts in this thread. If your question goes unanswered, it will probably be because the question has already been answered countless times in this thread.

This hack was created for use on my forums and will only be supported as time permits.


Links to required Perl Modules

• News::NNTPClient
• Net::SMTP
• MIME::WordDecoder
• Text::AutoFormat
• Mail::Address
• URI::Find
• DBI
• DBD::MySQL
• Date::Parse
• HTML::Entities

[fastforward]

The SMTP server won't affect things being sent to usenet. There are several things that may prevent it showing up on usenet:

For anyone having problems with newnews.pl, I've attached the one I am currently using at my site. It's been running against a dnews server without problems for several months.

1. An invalid email address. As far as I know, the NNTP server just requires an email that 'looks' like a valid email. However, there may be servers that require a working address.
2. You don't have posting rights on the server
3. The group doesn't exist on the server you are posting to.

There are other reasons why your post may not appear in the correct thread on usenet, but for it not to show up at all, it's usually one of the above reasons.

I'm currently re-writing the entire hack in php. There are major changes to the way it works:

• It will be specifically for vB 2.2.1 and php version 4.06 and above.
• I have written my own nntp class that can be used in other scripts. It's virtually a port of the perl one with a few extras.
• The news-pulling script will still be ran using a cron so you will need the php executable on your server. It may be possible to run it via the web, but no guarantees initially.
• There will be an option to retrieve only headers. These will be searchable as a normal post. When somebody clicks on the post, the script will download the bodies of all messages in that thread and display it as a normal post and thread. This option is best suited for people running the news server on their own servers where they can control how long a post is kept. This may be added after the initial release depending on time.
• The posting to usenet will be handled immediately by the 'newthread.php' and 'newreply.php'. The newnews.php will send any posts that may failed the above step, or were since moderated.
• The table structures have changed completely. The external usenet tables will be gone. Additional fields in the vB tables will handle everything. This reduces the volume of data and minimizes the chance of missing usenet posts.

There will be no upgrade path from previous perl versions.

[Scrooge]

For anyone having problems with newnews.pl, I've attached the one I am currently using at my site. It's been running against a dnews server without problems for several months.

An invalid email address. As far as I know, the NNTP server just requires an email that 'looks' like a valid email. However, there may be servers that require a working address.

You don't have posting rights on the server

The group doesn't exist on the server you are posting to.

I don't think any of those would cover my problem. Remember, my "quote" reply button works fine, just the "post reply" button does not seem to work. If any of the above was the problem wouldn't the "quote" reply fail as well?

[spideyrdr]

I've got a small problem with USENET posts getting stuck in the news server...

I'm working with a standalone inn news server and it looks like the posts COMING from my USENET server are off by exactly 1. If I post a message to the group then run newnews.pl, I get no new messages on the forum. If I post a second message or reply to the USENET post, then I get the original message, but not that new message. I wouldn't worry about it except I don't see a TON of my users going the USENET route, thus a message posted on USENET could get "stuck" for a while. Posts coming out of the forum end up in USENET fine. Any help would be greatly appreciated!

Thanks!

PS. I had to do another small mod to the perl to get messages to post to the inn news server... I had to set line ~ 1023 to "push(@article, "X-References: $threadid|$postid\n\n");" thus I needed an extra return in there. I'm surprised that only inn servers want that, but oh well.

[interrealm]

I'm running Windows2000 with IIS... is it possible to set this up? All references I've seen are for apache only.

[interrealm]

btt

[fastforward]

Originally posted by interrealm
I'm running Windows2000 with IIS... is it possible to set this up? All references I've seen are for apache only.

It can be done. There are posts in this thread detailing how.

You might want to wait a few weeks until I release the new PHP version that will not require the headache of installing any Perl modules.

[interrealm]

Sounds good to me... i have no problem waiting...

I would love to be a tester if you need one. my board currently sends gets around 1,200 to 1,300 post on the average daily.

[Scrooge]

I'm still having the same problem, post reply does not work but quote reply does. I just noticed that posting a new thread does not go to the usenet either. Anybody have a clue? Could FF's answers above be the cause after all? I still don't think so since posting with quote reply works correctly. Any help is GREATLY appreciated, I've been trying to fix this for a couple of weeks and really want to fix it so I can share Usenet with my users.

[fastforward]

Scrooge,

Did you install the latest newnews.pl I attached a few posts up?
Install that and if it still doesn't work, I'll see what I can do to help.

The newthread and new reply doesn't actually rely on templates for posting. It's only the quote that needs template changes... and that works.

The other two are simply extracted from the database when newnews.pl runs. It does this by checking a flag (isusenet) in the post and thread tables. This can be set -1, 0 or 1. 0 means it's a local post and will never be sent. -1 means it's already gone to usenet, and 1 means it's pending. Do a query on the post table and check if any are still set to 1.

The other point to remember is a reply to a post will only go to usenet if the post to which it is replying has a valid usenet message ID. If the parent post originated on the forum, it won't have this until we re-retrieve it from the news server on the next batch run. Replies will be held until that retrieval is made. This is why the useragent must remain static otherwise we can not identify which posts came from the forum.

Obviously, new threads do not suffer from this limitation as they do not have a parent.

Anyway... install the latest newnews.pl and report back

[Scrooge]

That did the trick, why didn't you say to do that before!? Oh, you did

Thanks a ton. This is the best hack out there IMHO. Now that everything is working I'm off to share the good news with my users.