NNTP Gateway for Usenet ( Newsgroups ), Mailing Lists

The latest version: V2.3.2 (31 Oct 2004)

What is it?

This hack provides a gateway from vBulletin forums to the USENET. It allows forum members to read newsgroup messages as well as to send posts to the newsgroups. In a nutshell, it turns vBulletin forums into a basic function news reader.

For an added bonus, this package also supports mailing list archive and forum to mailing list gateway. The software retrieves messages from a pop3 account and import them into the forums threaded. It also allows a forums user to send messages to the mailing list by the forum posting interface.

Why?

The gateway will be appreciated by forum members who do not know how to set up a news reader, or people who do not have port 119 open in their network. A forum interface to read and post news are often considered to be better than a news reader.

The gateway adds more contents to your forums and allows the members to interact with wider audience. It is common to see new users signing up to use this feature.

History

This hack is basically a hacked vB2 NNTP gateway written by Gilby to work with vB3. The original hack can be found here:

https://vborg.vbsupport.ru/showthrea...7&page=1&pp=15


Features

Some of the features and improvement include:

• Handles both newsgroup posts and mailing lists.
• Handles MIME encoded messages and attachment.
• Handles UUEncoded messages.
• Two Admin Control Panels for all settings.
• Automated installation.
• Correct thread implementation. All imported messages and posts sent to newsgroups will be threaded correctly according to References and Message-ID headers.
• Optional setting to thread by subject for mailing lists that strip references header.
• Supports TABLE_PREFIX.
• Imported messages will be searchable just like normal forum posts.
• Similar threads are built when the messages are imported if the system setting is on.
• Multiple attachments importing.
• Save attachment to database or file according to vB setting.
• Thumbnail created while importing image files.
• Multiple attachment post to newsgroup is also supported.
• Selectable X-No-Archive header honouring system.
• Light on system resources.
• Global Killfile to filter out unwanted messages. Killfiles can have OR and AND effectiveness.
• Strip footer by setting for incoming messages. Strips out repeated message footers.
• Built options to disable signatures and footers for out going messages.

Future development

• AdminCP interfaces. (completed)
• Installation/upgrade script. (completed, coded into AdminCP)
• Traditional "> " lead posting interface.

What do you need and do not need?

You do not need to edit vB php files if you use two bookmarks
You need to edit one vB php file to insert two lines of code if you want direct click links within AdminCP.
No need to run SQL.

Installation

Basically the AdminCP interfaces take care of installation as well as upgrade in the background. Back up your database first. The scripts can modify your database without alerting you!

Please follow the instructions in the readme.txt file for more details.

Support

Please post your support questions to this thread. Please check the FAQ in the third post within this thread first.

================== UPDATES ======================
Jul 31 2004 (release version 2.2.0)

• Provided option of not importing attachments
• Provided option of pause between messages
• Provided option of not sending messages to USENET
• Provided mechanism to skip importing a message if it was crashed on it
• Provided "Default Footer" option
• Provided strip_footer function for mailing lists
  (written originally by KevinM)
• Provided Killfile functions with Header/Body/AND/OR settings
• Provided ability to thread messages by subject for mailing lists
• Provided ability to use a single POP3 account for different mailing lists
  (use prefix)
• Merged KevinM's attachment routine, save thumbnail to hard drive
  as vB3.0.3
• Official support to have the script run by Scheduled Tasks
• Many bug fixes

Please note, when you manually run the script, you must turn on the debug mode. Otherwise nothing will be displayed on the screen.

To run as debug mode: http://forums.yourdomain.com/gateway.php?debug=1

25 Oct 2004

• Provided the option of sending posts using the real email address
• Provided the option of sending posts using a fixed email address for
  each group.
• Posts are now held back if they are in the moderating queue
• Scheduled task log is now working
• Provided the option of sending the debug messages to the vB3 cron log
• Bug fixes

31 Oct 2004

Fixed bugs running the gateway by the vB3 Scheduled Tasks

1 March 2004

Uploaded the new zip file. It contains one or two bug fixes.

[AWS]

Originally Posted by Imhotep

:ermm: My forums stopped grabbing new Usenet messages on 6/21. I'm trying to finger out exactly where the log jam has occured. My cron job is still going off like clockwork.

This is the entire content of my cron results minus the header garbage.

Gateway version 2.1.0  4 group(s) gatewayed.

Connecting to server, server says: 200 News.GigaNews.Com

Server responded after user name: 381 more authentication required

Server responded after password: 281 News.GigaNews.Com

Info for comp.graphics.apps.lightwave at news-40.giganews.com:
211 12589 237200 249788 comp.graphics.apps.lightwave

Getting message number 249690:

Any ideas as to what is happening? It looks like it only hits one group... then stops. ? :ermm:

Getting message number 249690:

The message is not there and the script thinks it has reached the end. Go to newsgroup settings and change the id number and run the script again.

[Imhotep]

awesome! thank you very much

[kmike]

Originally Posted by AWS

The message is not there and the script thinks it has reached the end. Go to newsgroup settings and change the id number and run the script again.

While you posted good workaround for this, the root of this problem is actually completely different. We encountered the same problem at about the same time, and the real cause was PHP crashing on the huge spam messages crossposted to several newsgroups. Here's an example (5000+ lines!):

http://groups.google.com/groups?dq=&...ing.google.com

The quick fix is to limit amount of lines we get from the news server per message.
After the line 128 of includes/nntp.php, which reads:

PHP Code:

        $post = ''; 


add this line:

PHP Code:

        $lc = 0; 


Then find this code fragment starting on line 134:

PHP Code:

            if ($line == ".") {
                break;
            } else {
                $post .= $line ."\n";
            } 


Change it to:

PHP Code:

            if ($line == ".") {
                break;
            } else if ($lc++ < 300) {
                $post .= $line ."\n";
            } 


Change 300 here to the amount of lines from news posting you want to allow to get into forum's message.

Enjoy

[Princeton]

A word (title or message) that long will break a fixed width forum. It will even make a liquid layout appear "bad".

The fetch_word_wrapped_string function does not work on long words with periods, commas, special characters. This is a bug that has been reported.

Until there's a fix ... you should use the fetch_trimmed_title on titles (showthread/forumdisplay). fetch_trimmed_title function works with the $vboptions[wordwrap] settings.

If you do not want to use the fetch_trimmed_title function...
I use this on my cms: (mainly for right/left column):

function gtp_strip_title($str, $start, $end) {
 	if (strlen($str) > $start){
      	$str = substr($str, 0, $end)."...";
 	}
      return $str;
 }

I suggest having an option to remove (not insert in db) messages with long titles.
All messages with long words (titles) are 100% spam.

[kkiely]

I am a newbie. With this script I keep hearing people saying to "run a cron job"

1. What is "cron"? I assume it is some type of scheduling task system.
2. What will it do with this script? does it auto-import the posts?

[Odoin]

A cron is where you go downtown and play the shell game and cron people out of their money...

Just kidding... many providers that offer a Control Panel have some way to setup a cron job. You are right what a cron is... It runs a script or program and a set time.

In this case you want it to run the gateway.php script to pick up and send new NNTP messages. You could do this your self by pointing your browser to the gateway.php file... however if you forget or are gone for a few days the new posts will not be updated. Another factor is on large NNTP groups it takes a lot more time if ran once a day rather then every hour.

[chopperp1]

Hello to All
I'm trying to get this hack working on company's web server. Everything was pretty much straight forward till I got to the point of running gateway.php. That's what I get:
"...pop3 gateway

Gateway version 2.1.0 1 group(s) gatewayed. Bad User Name!Retrieving Protocol E-mails from the 'dl360mail3.yamner.com' pop3 server ..."

I tripple checked all settings and everything looks good. User account for pop3 is correct including password. Web server talks to mail via local subnet (no filtering there). Can't figure out why this thing can't log in.
any clues? :ermm:

[lierduh]

Originally Posted by chopperp1

Hello to All
I'm trying to get this hack working on company's web server. Everything was pretty much straight forward till I got to the point of running gateway.php. That's what I get:
"...pop3 gateway

Gateway version 2.1.0 1 group(s) gatewayed. Bad User Name!Retrieving Protocol E-mails from the 'dl360mail3.yamner.com' pop3 server ..."

I tripple checked all settings and everything looks good. User account for pop3 is correct including password. Web server talks to mail via local subnet (no filtering there). Can't figure out why this thing can't log in.
any clues? :ermm:

The best way is to log into your web server and do a telnet from there:

#telent dl360mail3.yamner.com 110
#USER your-username
#PASS your-password
#STAT

I suspect your pop3 server does not like the pop3 access from a local subnet. You gave a public domain name, yet accessing through a local private net. Either routing or pop3 server's "allowed access" can cause your problem.

[john_rsd]

Ive been testing this for a few days now.

I have found that the script has crashed twice causing the following error

Database error in vBulletin 3.0.1:

Invalid SQL: INSERT INTO attachment SET dateline = '1088526838',
filename = '',
filedata = 'GIF89a?\0,\0?\0

<<snipped here>>
thumbnail = '',
filesize = + 0,
postid = 202
mysql error: You have an error in your SQL syntax. Check the manual that corresponds to your MySQL server version for the right syntax to use near '+ 0,
postid = 202' at line 5

mysql error number: 1064

Date: Tuesday 29th of June 2004 09:33:59 AM
Script: http://proteluser.com/bbs3/gateway.php
Referer:
Username: Site_Admin

But the script seems to have went so far then went pear shaped, at this point threading by title was completely non-effective and the 22 messages were not threaded at all despite having subject headers matching exactly. See screenshot attached.

There also seems an issue with handling possible EOL characters as i got a message like this inserted to the forums. Notice the '=' inserted at the end of a line before a return.

I've just been able to use DXP for a complete job and feel that I can now c=
omment on a few things that I've found.

Of course some of this may have a new 'DXP' way of doing things that I don'=
t know of yet.

In 99Se I could 'RMB' cut, copy or paste when editing many parameters such =
as drawing number in the schematic editor, I can't do this anymore and inst=
ead must use the more cumbersome ctrl C / ctrl V method.

Usually when editing text I want to change the text itself - in 99SE, doubl=
e clicking on a text string opened an editor panel with the 'text' selected=
.. This was very good for text editing as all I needed to do was to double c=
lick, type in the new value and then enter. Now I double click and then hav=
e to find and select the 'value' to change.

It should have looked like this, at least this is how outlook displays it in the mail box.

I've just been able to use DXP for a complete job and feel that I can now comment on a few things that I've found.

Of course some of this may have a new 'DXP' way of doing things that I don't know of yet.

In 99Se I could 'RMB' cut, copy or paste when editing many parameters such as drawing number in the schematic editor, I can't do this anymore and instead must use the more cumbersome ctrl C / ctrl V method.

Usually when editing text I want to change the text itself - in 99SE, double clicking on a text string opened an editor panel with the 'text' selected. This was very good for text editing as all I needed to do was to double click, type in the new value and then enter. Now I double click and then have to find and select the 'value' to change.

This message was sent as plain text.

This was the second message inserted in that batch. The first message is uneffected and was threaded correctly.

Is there any compatability issues with mysql or php versions ???

Any help appreciated

[john_rsd]

After looking at the messages retreived from the pop3 mailbox on a different account to the one used for gatweway.php i noticed that there was a few messages originally in HTML and with gif as attachments.

Could the above error be an issue with trying to insert the attachment?

Attachments are usually frowned upon on mailing lists which is why I failed to notice it when making my last post.

Is there a quick fix to simply drop attachments completely?