[RELEASE] Usenet Gateway for vB 2.01

Description:

This hack provides a gateway to selected usenet groups. The entire usenet group is mirrored locally. Posts submitted locally to mirrored newsgroups will also be sent out to usenet. A single perl script is executed via cron that collects news and posts any outgoing articles. Incoming messages are parsed against customizable spam filters and threaded correctly before being placed in your forums. By default, no personal information is sent to usenet other than the posters username and whatever you have globally configured as a footer and organization. The option does exist for a user to include a custom email to use for usenet posts. After initial installation (which involves creating some new tables, adding a few columns to existing vB tables, and entering the database connection information to the news script) configuration is completed via the vB control panel.

• See it in action at dBforums.com (usenet forums are at the bottom)
• Download latest version for vB 2.2.4

Current version is 20020323 for vB2.xx

Current feature list:

• correct threading of outgoing posts on usenet
• full control panel integration
• email notification to usenet replies
• emoticon translation into vb icons
• vb code removal or conversion prior to posting to usenet
• hyperlinked urls in messages
• color coded, italicized & indented quotes
• vB style quote to usenet style quote conversions for outgoing posts
• logging of outgoing posts
• support for multiple news servers
• support for seperate footers per forum in outgoing posts
• multi-language (selectable) handling of quoted MIME printable headers (for all those funny foreign characters)
• handling of mailing archive groups
• flexible spam control and replacement variable options for incoming and outgoing messages
• configurable auto-expire option
• option for users to show email address in their outgoing usenet posts (may be different than normal one in profile
• option to enable/disable user signatures on outgoing posts.
• fully compatible with vB moderation functions

Still to be done:

• canceling of messages after they have been sent to usenet via control message in accordance with RFC1036.
• binary attachment support
• email to PM gateway

Screeshots:
Main Options
More Options
Configuration of Newsgroups
Add New Newsgroups
Spam Control
What an imported usenet article looks like in vB

The latest version of this hack will always be in this first post of the thread.

Before asking questions in this thread, please read all the posts in this thread. If your question goes unanswered, it will probably be because the question has already been answered countless times in this thread.

This hack was created for use on my forums and will only be supported as time permits.


Links to required Perl Modules

• News::NNTPClient
• Net::SMTP
• MIME::WordDecoder
• Text::AutoFormat
• Mail::Address
• URI::Find
• DBI
• DBD::MySQL
• Date::Parse
• HTML::Entities

[fastforward]

Ah.. I see. Yeah that could be a nice option, and a lot easier to implement than trying to encode attachments. I'll definately add that.

[jOOPeON]

Originally posted by fastforward
Ah.. I see. Yeah that could be a nice option, and a lot easier to implement than trying to encode attachments. I'll definately add that.

Thank you !

Please let me know when the modification is ready for test.

thanks in advance,

-jOOP

[Gilby]

In the php version of the hack I wrote, I just made a link at the bottom of the post that had the url to the attachment and listed the filename. That method seems to work good and allows my users to post pictures or whatever and it's viewable on the text-only newsgroup as well. Fastforward, you may want to look at how that's done.

[jOOPeON]

Something strange happened today.
One of usenet imports failed to the complete crash of perl.
This happened over and over again, until I noticed that it happens only when importing replys from one of the groups.
When I disabled that group, all another groups imported succesfully. If I enable the "broken" group again, same thing happened.
Newnews just crashes when " loading replies 2" .
I tried to increase the last message number wíthout any help.

What could be wrong?

During translations I'm temporary running localhost, IIS 5, Advanced Server, vB 2.2.3, newest perl and PHP ( going back to linux for sure )

-jOOP

ps. I could include the dump from Windows, but it may not help?

[etom]

Hi.

Just installed this great peace of software .

Fetching (the first) 500 articles from my newsserver was done in 2 minutes, but inserting threads and loading replies needed 3 hours !!

I want to import 40000 posts from my server to vb, but at this speed ?

What to do or to look for ?

PS: my vb is currently at 700 posts.

Thanks
etom

[fastforward]

Well that obviously doesn't sound right

Even with a low spec machine it shouldn't take anywhere near that long. As an example, my single processor PIII750 inserts a batch of 500 into the database in 2 or 3 minutes. This is with just under a million posts.

There are few things you should check.
1) The Perl version and it's modules. The Perl version should be 5.005_03 or higher and make sure you have the latest modules.

2) Make sure all indexes are in place in the database. Critical indexes are on the msgid columns and the refs columns. Also make sure all the original vB indexes are in place.

At what point is the delay occuring? What is displayed on the screen?

[etom]

Shell output:
...
-> fetching 23941... OK
-> fetching 23942... OK
-> fetching 23943... OK
-> processing article batch...
-> requested 500 messages... 0 not available or rejected.
-> inserting new threads from alt.x.y.z
[this needed 2 hours.]
*-> Post 2667 skipped... (too long)
-> inserted 113 threads
-> finding replies...
-> loading 367 replies...
[waiting for 4 hours here (currently at 455 = will finish soon)]

My hardware:
Its a 1000 Mhz AMD and a remote 1400 Mhz AMD Database Server, not much load on both.

No changes to default indexes were made ... do You have a reference structure to compare with ?

PS: Thank You very much for Your fast help

etom

[Stephan Whelan]

etom,

I'm almost positive you'll be missing an index somewhere as I had the exact same problem when I first installed the software. I ran with it for around 2 months before figuring out it there was a significant problem!

[fastforward]

Here's what the indexes should be.

POST
Unique
msgid

Non-Unique
isusenetpost


THREAD
Unique
msgid

Non-Unique
isusenetpost

Non-Unique
regpost


USENET_ARTICLE
Primary key
msgid
forum


USENET_ARTICLE_LOADER
Primary Key
msgid

Non-Unique
forum


USENET_REF
Primary Key
msgid
ref


USENET_REPLACE
Primary Key
id

Unique
original


USENET_SETTING
Primary Key
settingid


USENET_SPAM_FILTER
Primary Key
id

Non-Unique
forum
dtm


USENET_GROUP
Primary Key
newsgroup


USENET_OUTGOING
Primary Key
postid

Non-Unique
ref


USENET_SETTINGGROUP
Primary Key
settinggroupid


Unique
expression

[fastforward]

An update on what's happening with the next version:
I've OOPed everything and split everything into modules and classes.

There will be the following modules:
NNTP - A slight extension to the standard Perl module
MySQL - Database functionality
Article - A class for formatting and processing usenet articles
vB224 - An interface to vBulletin, handling inserts, indexing etc
Config - Script configuration class

These modules will all be called from a single script that controls the program execution.

The forum module (vb224.pm) will have a standard interface that will allow different versions of vB (or even other forums) to be used by simply dropping in the correct module. Basicaly, this module will be passed an instance of the Article class.

This new version should still be a drop in replacement to newnews.pl.

I have completed the first cut of all modules except vB224 (although, I'm well on the way to finishing it). I'm away on vacation next week so I won't be working on it until I get back.