[RELEASE] Usenet Gateway for vB 2.01

Description:

This hack provides a gateway to selected usenet groups. The entire usenet group is mirrored locally. Posts submitted locally to mirrored newsgroups will also be sent out to usenet. A single perl script is executed via cron that collects news and posts any outgoing articles. Incoming messages are parsed against customizable spam filters and threaded correctly before being placed in your forums. By default, no personal information is sent to usenet other than the posters username and whatever you have globally configured as a footer and organization. The option does exist for a user to include a custom email to use for usenet posts. After initial installation (which involves creating some new tables, adding a few columns to existing vB tables, and entering the database connection information to the news script) configuration is completed via the vB control panel.

• See it in action at dBforums.com (usenet forums are at the bottom)
• Download latest version for vB 2.2.4

Current version is 20020323 for vB2.xx

Current feature list:

• correct threading of outgoing posts on usenet
• full control panel integration
• email notification to usenet replies
• emoticon translation into vb icons
• vb code removal or conversion prior to posting to usenet
• hyperlinked urls in messages
• color coded, italicized & indented quotes
• vB style quote to usenet style quote conversions for outgoing posts
• logging of outgoing posts
• support for multiple news servers
• support for seperate footers per forum in outgoing posts
• multi-language (selectable) handling of quoted MIME printable headers (for all those funny foreign characters)
• handling of mailing archive groups
• flexible spam control and replacement variable options for incoming and outgoing messages
• configurable auto-expire option
• option for users to show email address in their outgoing usenet posts (may be different than normal one in profile
• option to enable/disable user signatures on outgoing posts.
• fully compatible with vB moderation functions

Still to be done:

• canceling of messages after they have been sent to usenet via control message in accordance with RFC1036.
• binary attachment support
• email to PM gateway

Screeshots:
Main Options
More Options
Configuration of Newsgroups
Add New Newsgroups
Spam Control
What an imported usenet article looks like in vB

The latest version of this hack will always be in this first post of the thread.

Before asking questions in this thread, please read all the posts in this thread. If your question goes unanswered, it will probably be because the question has already been answered countless times in this thread.

This hack was created for use on my forums and will only be supported as time permits.


Links to required Perl Modules

• News::NNTPClient
• Net::SMTP
• MIME::WordDecoder
• Text::AutoFormat
• Mail::Address
• URI::Find
• DBI
• DBD::MySQL
• Date::Parse
• HTML::Entities

[fastforward]

Originally posted by Dioxin
thanx for your answer, fastforward

i have another one:

there are still many many threads awaiting for manual post insertion. is this normal? how can i avoid this?

what's the meaning of orphan? i'am a german guy and i don't understand this option in control panel. should i set it to 5 or a higher level.

thanx,

regards, dioxin

This question comes up time after time. Here are the reasons:

Posts are inserted into the forums if and only when these two conditions are true:

1. a post with a msgid that appears in the list of references in the header of the new article is already in the forum.
2. the ord column of the new article is greater than or equal to the ord column of the existing post in the forum.

There are various reasons why you may have orphan replies with no where to go:

• the thread starter has already been expired from the news server and it was never retrieved.
• the thread starter has already been expired from your forums based on your control panel settings.
• the thread starter was rejected by your news server due to spam
• the thread starter was rejected by your spam filters
• the thread starter was rejected due to exceeding the maximum message length you have set.

The orphan setting is simply there to delete these orhan records from the usenet_article table. If the post to which the article replies has not appeared after 2 days, you can pretty much guarantee it's never going to arrive. The only reason it is configurable is to allow a full initial historical news pull that may go back several months.

[DarkReaper]

I still need help, here's the problems I had:

I got loads of errors when running those 2 SQL scripts.
Somewhere during the install, my whole boards got screwed. newthread.php was showing a blank screen except for the ad(which is in the phpinclude) and so was newreply, but they still posted threads/replies. However, when posting, it didn't show up as a new thread in the forum stats(threads, replies...), it didn't change the user's post count, and new replies didn't show up under the thread stats(replies, views..) but views did.

What am I doing wrong?? Or is it something wrong with the script?

[fastforward]

Originally posted by DarkReaper
I still need help, here's the problems I had:

I got loads of errors when running those 2 SQL scripts.
Somewhere during the install, my whole boards got screwed. newthread.php was showing a blank screen except for the ad(which is in the phpinclude) and so was newreply, but they still posted threads/replies. However, when posting, it didn't show up as a new thread in the forum stats(threads, replies...), it didn't change the user's post count, and new replies didn't show up under the thread stats(replies, views..) but views did.

What am I doing wrong?? Or is it something wrong with the script?

The script does nothing that affects your php pages. The errors can only be a result of your edits. I suggest reverting all your php files back to the originals and making each change individually. In fact, don't make any changes until you have the news coming in.

[DarkReaper]

I reverted the files back to the original ones, and still got the error. I'm assuming it has something to do with the database modifications. I got lots of errors when trying to install those, and corrected them, but something could have screwed up. Its really weird though. The error is there, but when I hit refresh it goes away.

[fastforward]

Originally posted by DarkReaper
I reverted the files back to the original ones, and still got the error. I'm assuming it has something to do with the database modifications. I got lots of errors when trying to install those, and corrected them, but something could have screwed up. Its really weird though. The error is there, but when I hit refresh it goes away.

The database changes shouldn't affect anything (at least not with those symptoms).

It could be you have cached pages somewhere (locally or a remote proxy). The cached pages are from when you were getting the errors. The refresh obviously forces a new page and the corrected file is retrieved.

There is a funny "bug" in the newest version, although, since I was going through all the different perl versions I'm not quite sure if it's not the perl modules that should be blamed; one thing is certain - previous versions of Usenet hack on my board didn't do it:

I've noticed (on my own example) that posts sent from news client (not the board) with nick names in square brackets (good example - my usenet nick is [MRK]) are displayed on the board either as Anonymous or using first part of e-mail before '@' sign. Any idea why?

[fastforward]

Originally posted by v0n
There is a funny "bug" in the newest version, although, since I was going through all the different perl versions I'm not quite sure if it's not the perl modules that should be blamed; one thing is certain - previous versions of Usenet hack on my board didn't do it:

I've noticed (on my own example) that posts sent from news client (not the board) with nick names in square brackets (good example - my usenet nick is [MRK]) are displayed on the board either as Anonymous or using first part of e-mail before '@' sign. Any idea why?

Email addresses are extracted from the 'from' line using a different module than the previous script versions. It now uses Mail::Address whereas it used to use Email::Find. It appears it simply ignores anything in the square brackets. I'll see what I can do about it.

[fastforward]

Release 20010721 fixes a few minor problems with newnews.pl.

• Fixed the problem of angled brackets being shown as hashes for the 'from' line of usenet posts.
• Fixed a problem with the article loader that may have caused duplicates if the message counters got reset.
• Fixed the possibility of endless looping problem when re-importing local threads.
• Simplified article batch pull routine.
• Cleaned up console messages.
• Put in a work-round for the Mail::Address problem with square brackets.
• Fixed problem with signature going out even though the outgoing post is set to not show it.

To upgrade, you just need to replace your existing newnews.pl with the one in this package.

I'm off on vacation today (Saturday), so I won't be around for the next week. byeeee!

The latest version of this hack will always be in this first post of the thread.

I'm thinking of developing a site centering around a vBulletin and especially using this great Usenet hack.

My question is how big does the database get with all of the usenet posts? Are we talking up to a couple gigs, or into the tens and hundreds?

I know it's all variable, depending on the number of Usenet posts, I was just curious what to expect.

Thanks,
Cameron

Greetings:

Well, I've been trying to figure this out for a week, so I guess I'll start begging for help here =)

I've got the newsgroups importing just fine, I have over 100,000 old usenet posts archived now, and new ones are filtering in every day. Everything in this department works great.

Where the problem comes, is outgoing. I have a dedicated news server here, and I can see that it's never making any attempt to send outgoing threads and replies. Looking in my database, the two outgoing fields are always empty, etc. It's simply not processing any posts that I make to the newsgroup forums as outgoing.

Any ideas? I have the latest version of everything required, running under apache with php as isapi, and it's running on a dedicated win 2000 box.