[RELEASE] Usenet Gateway for vB 2.01

Description:

This hack provides a gateway to selected usenet groups. The entire usenet group is mirrored locally. Posts submitted locally to mirrored newsgroups will also be sent out to usenet. A single perl script is executed via cron that collects news and posts any outgoing articles. Incoming messages are parsed against customizable spam filters and threaded correctly before being placed in your forums. By default, no personal information is sent to usenet other than the posters username and whatever you have globally configured as a footer and organization. The option does exist for a user to include a custom email to use for usenet posts. After initial installation (which involves creating some new tables, adding a few columns to existing vB tables, and entering the database connection information to the news script) configuration is completed via the vB control panel.

• See it in action at dBforums.com (usenet forums are at the bottom)
• Download latest version for vB 2.2.4

Current version is 20020323 for vB2.xx

Current feature list:

• correct threading of outgoing posts on usenet
• full control panel integration
• email notification to usenet replies
• emoticon translation into vb icons
• vb code removal or conversion prior to posting to usenet
• hyperlinked urls in messages
• color coded, italicized & indented quotes
• vB style quote to usenet style quote conversions for outgoing posts
• logging of outgoing posts
• support for multiple news servers
• support for seperate footers per forum in outgoing posts
• multi-language (selectable) handling of quoted MIME printable headers (for all those funny foreign characters)
• handling of mailing archive groups
• flexible spam control and replacement variable options for incoming and outgoing messages
• configurable auto-expire option
• option for users to show email address in their outgoing usenet posts (may be different than normal one in profile
• option to enable/disable user signatures on outgoing posts.
• fully compatible with vB moderation functions

Still to be done:

• canceling of messages after they have been sent to usenet via control message in accordance with RFC1036.
• binary attachment support
• email to PM gateway

Screeshots:
Main Options
More Options
Configuration of Newsgroups
Add New Newsgroups
Spam Control
What an imported usenet article looks like in vB

The latest version of this hack will always be in this first post of the thread.

Before asking questions in this thread, please read all the posts in this thread. If your question goes unanswered, it will probably be because the question has already been answered countless times in this thread.

This hack was created for use on my forums and will only be supported as time permits.


Links to required Perl Modules

• News::NNTPClient
• Net::SMTP
• MIME::WordDecoder
• Text::AutoFormat
• Mail::Address
• URI::Find
• DBI
• DBD::MySQL
• Date::Parse
• HTML::Entities

[NgtCrwlr]

Greets Paul,

Well I finally got motivated enough to update vB from 2.0.3 to 2.2.5. I also updated to the latest newnews.pl and everything seems pretty sweet except for a little line feed problem.

On both (NewThreads) and (Reply posts) the messages display fine locally, but the posts to USENET lose the linefeed formatting on any line that ends with a single carriage return.

Example:

line 1
line 2
line 3

line 4 (space between line 3 and 4)
line 5


And this is the end double spaced from line 5.

Will display on USENET as:

Line 1 Line 2 Line 3

Line 4 (space between line 3 and 4) Line 5


This should be double spaced from line 5.

Everything else seems to be working with vB 2.2.5 very nicely. You have my vote for continuing the Perl version of the hack. I hope that things are going well on your end.

Cheers!

[ixian]

Fastfoward, I know this hack has been out for a while and you've probably heard this before, but I just wanted to say THANKS. This hack ROCKS. I am in the process of overhauling my entire site, including forums, and made this the first hack I installed on my new vb 2.2.5 installation after I imported all my users, posts, threads, etc over. It's working like a charm in the new forums now (they don't go live until next week).

I suspect my users are gonna go nuts for this hack, so thanks again! BTW, I had zero problems (other than a misconfigured CPAN config.pl, but I fixed that and anyway that isn't the hacks fault ) installing this hack - there's something to be said for a fresh vb install.

P.S. I've done a few test posts and I don't see the linefeed problem mentioned above w/ 2.2.5. Maybe it's something with the upgrade from 2.03? I installed 2.2.5 on a new server from scratch.

[www.tourc.ca]

fastforward: I must say this is by far the best hack on my site, and had very few problems that weren't sorted out right away, and the documentation is pretty straight forward. Thanks very sincerley for doing this hack and supporting it so damn well! I'll be sticking with the perl version myself! so far there isn't a problem I couldn't solve with a little help from google!

I'm a little new to cron, and I've read this whole thread (all 32 pages) and haven't found an answer. I do hope you get your benz... you definatley deserve it in my books! I've also read tons of pages on using cron, and now understand it!

I've created a crontab (crontab -e) and a job 10,30,50 * * * * perl /home/admin/newnews.pl and so far it works and e-mails the output to me, but it has problems finding some perl modules.

if I run the command perl newnews.pl from the prompt via ssh the script runs beginning to end without a problem! it's perfect, problem is I want to obviously have this automated, and cron's the tool, but it seems it can't find the perl modules when cron runs it!

the first time it ran, I got the following message:
Can't locate News/NNTPClient.pm in @INC (@INC contains: /usr/lib/perl5/5.6.0/i386-linux /usr/lib/perl5/5.6.0 /usr/lib/perl5/site_perl/5.6.0/i386-linux /usr/lib/perl5/site_perl/5.6.0 /usr/lib/perl5/site_perl .) at /home/admin/newnews.pl line 42. BEGIN failed--compilation aborted at /home/admin/newnews.pl line 42.

somewhere else in this thread (as I assumed from the message) it states that it can't find the path. I thought it was a little strange that it works fine when I run it manually and not through cron, but I uncommented the line in the newnews.pl script that is:

# uncomment the following line if you installed perl modules to a
# non-standard directory
use lib "/usr/local/lib/perl5/site_perl/5.6.1";

note: the NNTPClient.pm was located (locate NNTPClient.pm) in /usr/local/lib/perl5/site_perl/5.6.1/News/NNTPClient.pm

now it can find the NNTPClient.pm file but now I get the following message and the script stops on the line
use MIME::WordDecoder;

Can't locate MIME/Base64.pm in @INC (@INC contains: /usr/local/lib/perl5/siet_perl/5.6.1/MIME/Decoder /usr/local/lib/perl5/site_perl/5.6.1 /usr/lib/perl5/5.6.0/i386-linux /usr/lib/perl5/5.6.0 /usr/lib/perl5/site_perl/5.6.0/i386-linux /usr/lib/perl5/site_perl/5.6.0 /usr/lib/perl5/site_perl .) at /usr/local/lib/perl5/site_perl/5.6.1/MIME/Words.pm line 85. BEGIN failed--compilation aborted at /usr/local/lib/perl5/site_perl/5.6.1/MIME/Words.pm line 85. Compilation failed in require at /usr/local/lib/perl5/site_perl/5.6.1/MIME/WordDecoder.pm line 77. BEGIN failed--compilation aborted at /usr/local/lib/perl5/site_perl/5.6.1/MIME/WordDecoder.pm line 77. Compilation failed in require at /home/admin/newnews.pl line 45. BEGIN failed--compilation aborted at /home/admin/newnews.pl line 45.


i typed 'locate Base64.pm' and it returned

/usr/local/lib/perl5/site_perl/5.6.1/i686-linux/MIME/Base64.pm
/usr/local/lib/perl5/site_perl/5.6.1/MIME/Decoder/Base64.pm

i tried adding another use lib line below the first in the perl script but cannot get it past this error. I used the following format!

use lib "/usr/local/lib/perl5/site_perl/5.6.1";
use lib "/usr/local/lib/perl5/site_perl/5.6.1/i686-linux";

but this isn't working and I continue to get that last message about not being able to find base64.pm

I'm new to most of this and have tried a ton of things and am kind of stuck here!... Please help, and forgive my ignorance!

dave

[fastforward]

I'm back... in case anyone missed me

Sorry I've not responded to this thread for a while. I went through one of those phases when you switch focus to other interests. Web stuff seemed to take a back seat. I'm sure we all go through that now and again.

www.tourc.ca, I've responded via PM to your problem. You should be able to add multiple 'use lib' statements to the script.

NgtCrwlr, I seem to remember a similar problem with line feeds way back. It may simply changing all occurances of '\n' with '\r\n' or something similar. I'll look into it.

I still plan on the major script overhaul for vB version 3.

[NgtCrwlr]

Originally posted by fastforward
Sorry I've not responded to this thread for a while. I went through one of those phases when you switch focus to other interests. Web stuff seemed to take a back seat. I'm sure we all go through that now and again.

No problemo. In the last year my interest in the web has certainly taken a much lower priority too (most of the time dead last). So you are not rowing the boat alone.

Cheers!

-Larry
P.S. You have been missed.

[asarian]

"Upload the newnews.pl script into any directory and chmod 0755 to make it executable."

Any reason why it needs to be chmod 755? It has username + password hardcoded in it. Lets just make it chmod 700, and we will all feel a lot safer.

- Mark

System Administrator Asarian-host.org

---
"If you were supposed to understand it,
we wouldn't call it code." - FedEx

[asarian]

If I may be so bold as to make a tiny suggestion, I would like to change the way newnews.pl now brutally kills (kill -9) when a previous instance is running.

For other "new-news" utilities, I use the following code at the beginning of my prog,

open (NEWSLOCK, '>/tmp/new-news.lock') || exit 1;
exit 1 if (not (flock (NEWSLOCK, 6)));

That will try to acquire an exclusive lock, but leave when it cannot immediately do so. That way, when the previous instance has not finished yet, nothing needs to be brutally killed (jn whatever state of update it might be), and instead this ensures no new update begins until the last one ended.

- Mark

System Administrator Asarian-host.org

---
"If you were supposed to understand it,
we wouldn't call it code." - FedEx

[www.tourc.ca]

crontab is working and the hack is running flawlessy. Simply Amazing! I love it.

just a thought on a feature:
having seperate expire days value for each group

this would be great for those groups that get very little posts, but have good information that would be worth saving for later reference, you could have a high expire day value for them, but groups with huge amounts of posts could be set with a lower expire date value!

just a thought!

has anyone run a larger number of groups with this hack and had indexing enabled!

I tested it with indexing enabled and my searchindex table started getting really large, so its disabled for now, but I'm really missing being able to search the newsgroups through my forums.

does anyone have any kind of aprox. statistics of their site and how big their searchindex tables get and how it might effect performance and/or any other issues that might arise!

hmmm.. now the mundane task of sifting through the groups for spam and making spam filters... yeah!... I'm sure I'll be back with some questions regarding this!

great hack fastforward!

if anyone is currently using this hack on their forums, post the link I wouldn't mind taking a look at how you set it up (template wise). I'm running a seperate template as recommended by fastforward, and would like some ideas. (I love the dbforums look).

if you want to see the usenet gateway working on a site other than www.dbforums.com you can see it at www.tourc.com and use the username: testuser with the same password!

cheers!

note: to anyone first installing this hack be warned, when I was installing and searching the newsgroups I came across a group where someone from another forum was trying to install the same hack, he was successfull in doing so, but during his first run of the script not only did he pull all of the news articles, but he posted all of the thousands of threads he had in that forum to the newsgroup. He got flamed pretty bad, and it is affecting the usefullness of his forums on usenet... It's probably a good idea to create a new group for each newsgroup and not use existing forums! just a warning!

[Dmitriy]

I wonder what are the legal issues of copying posts from public newsgroups and posting them on your own website.

Google is doing the exact same thing at http://groups.google.com/ but does it mean that we can do the same thing too or does google have permissions from all the newsgroups that they allow browser-based access to?

[ixian]

Please don't take this as insulting, because it isn't meant to be, but, you need to seriously read up on what USENET is because I can't understand why this would even be a question.

Here's your first hint - who would sue you and why?