[RELEASE] Usenet Gateway for vB 2.01

Description:

This hack provides a gateway to selected usenet groups. The entire usenet group is mirrored locally. Posts submitted locally to mirrored newsgroups will also be sent out to usenet. A single perl script is executed via cron that collects news and posts any outgoing articles. Incoming messages are parsed against customizable spam filters and threaded correctly before being placed in your forums. By default, no personal information is sent to usenet other than the posters username and whatever you have globally configured as a footer and organization. The option does exist for a user to include a custom email to use for usenet posts. After initial installation (which involves creating some new tables, adding a few columns to existing vB tables, and entering the database connection information to the news script) configuration is completed via the vB control panel.

• See it in action at dBforums.com (usenet forums are at the bottom)
• Download latest version for vB 2.2.4

Current version is 20020323 for vB2.xx

Current feature list:

• correct threading of outgoing posts on usenet
• full control panel integration
• email notification to usenet replies
• emoticon translation into vb icons
• vb code removal or conversion prior to posting to usenet
• hyperlinked urls in messages
• color coded, italicized & indented quotes
• vB style quote to usenet style quote conversions for outgoing posts
• logging of outgoing posts
• support for multiple news servers
• support for seperate footers per forum in outgoing posts
• multi-language (selectable) handling of quoted MIME printable headers (for all those funny foreign characters)
• handling of mailing archive groups
• flexible spam control and replacement variable options for incoming and outgoing messages
• configurable auto-expire option
• option for users to show email address in their outgoing usenet posts (may be different than normal one in profile
• option to enable/disable user signatures on outgoing posts.
• fully compatible with vB moderation functions

Still to be done:

• canceling of messages after they have been sent to usenet via control message in accordance with RFC1036.
• binary attachment support
• email to PM gateway

Screeshots:
Main Options
More Options
Configuration of Newsgroups
Add New Newsgroups
Spam Control
What an imported usenet article looks like in vB

The latest version of this hack will always be in this first post of the thread.

Before asking questions in this thread, please read all the posts in this thread. If your question goes unanswered, it will probably be because the question has already been answered countless times in this thread.

This hack was created for use on my forums and will only be supported as time permits.


Links to required Perl Modules

• News::NNTPClient
• Net::SMTP
• MIME::WordDecoder
• Text::AutoFormat
• Mail::Address
• URI::Find
• DBI
• DBD::MySQL
• Date::Parse
• HTML::Entities

[AJR]

The problem I have above is with the 'TRUNCATE' MySQL command. If I try and run the command 'TRUNCATE TABLE usenet_article_loader' it comes back with a MySQL error. Is this a valid command for MySQL? If so, what version of MySQL supports it? I've been to the MySQL manual website and from what I understand, TRUNCATE basically drops and re-creates a table wiping out all data in that table but keeping the table structure. It's supposed to be more efficient than deleting each record individually.

My question is since the 'TRUNCATE TABLE usenet_article_loader' doesn't work on my server(s) that I've tested it on, will the 'DELETE FROM usenet_article_loader' do the same thing? Can I change the queries in the newnews.pl file from TRUNCATE TABLE' to 'DELETE FROM' ??

Right now what I'm doing is I remmed out the 'TRUNCATE TABLE' command from the newnews.pl file so it doesn't error out when running from the telnet command. After each group of messages is received (500 at a time), I'm going into phpMyAdmin and Emptying the usenet_article_loader table, then running the next batch of 500.

Also one other question... when the newnews.pl file gets run, I have set it to pull articles with less than 100 lines for right now. How long should it be taking to pull 500 messages? It seems REALLY slow (15 seconds/article). I use mammothnews.net and in my Agent Newsreader, it pulls 1500 article headers in about 20 seconds! So I know it's not the mammoth server. It did run a lot faster on Friday when I first got the TRUNCATE error (2 seconds/article). I have 11,000 articles to pull and it will take a really long time to get my news up to date!

Any help is appreciated...I would like to get this hack working completely and running on my site.

[fastforward]

You can replace TRUNCATE with DELETE. There won't be any problems (except slight speed impact).

As for the speed of articles being pulled; you should be getting between 1 and 3 articles per second. If you are sure it's not your news server, then it's probably something to do with the combinations of version of Perl and the modules. Not much you can do if you don't have full control over which versions you can install.

[KimG]

Hi,
I have just installed this great hack; after waiting many months for my host to switch me to my own server, and then install the modules!

Seems to work fine. I just have a couple of questions:

1) Spam control. Is it possible to only retrieve messages if they contain a certain word? How would I do this?

2) I have noticed that dbforums and other sites running this hack have the section of the vbulletin home page layout differently to other sections - ie. no moderators and differnt styles. How do you do this. Please could you just point me in the right direction!

Great hack - you should have charged

Kim

[AJR]

KimG... I'm working on changing the layout for my usenet group forums to be different than the normal forums. You have to add a new Style and Template set (I called them 'usenet'). Then change the style to use the 'usenet' templates. Then modify the forums to use the 'usenet' style. Hope this helps!

[KimG]

Originally posted by AJR
KimG... I'm working on changing the layout for my usenet group forums to be different than the normal forums. You have to add a new Style and Template set (I called them 'usenet'). Then change the style to use the 'usenet' templates. Then modify the forums to use the 'usenet' style. Hope this helps!

Thanks - would that alter the way that the newsgroups are displayed on the main forums page though. I just want a section at the bottom of my forums homepage (index.php) which list the newsgroups, maybe without the moderator columns etc.

[bitbender]

Originally posted by bitbender
I have some questions:

1. The expiration routines. How do they work? Under what criteria do they expire threads ?
2. Should usenet posts be threaded into forums of the same name for easy segmentation? (like COMP.OS.LINUX.SETUP)
3. If I start a thread in one of the segregated usenet groups, will it thread it out to the named usenet group for reply ??

Thanks!

???

[fastforward]

Originally posted by bitbender

The expiration routines. How do they work? Under what criteria do they expire threads ?

Should usenet posts be threaded into forums of the same name for easy segmentation? (like COMP.OS.LINUX.SETUP)

If I start a thread in one of the segregated usenet groups, will it thread it out to the named usenet group for reply ??

The expire is done on 'number of days old', configurable via control panel.

You don't have to name the forum the same as the newsgroup. The mapping is done via the control panel. You can assign any forum to contain usenet posts.

[fastforward]

Originally posted by KimG
1) Spam control. Is it possible to only retrieve messages if they contain a certain word? How would I do this?

No, not possible and there are no plans to add it.

[fastforward]

Originally posted by fastforward

No, not possible and there are no plans to add it.

Actually, you can do it. Just create a regular expression that rejects everything except the ones you want. You'll need to read up on regular expressions to get it right.

Is there a simple test to confirm if you have nntp installed on your box or not? Would prefer to know before I attempt to install one.

Man, a thread search would be nice on a thread this long uh?