NNTP Gateway for Usenet ( Newsgroups ), Mailing Lists

The latest version: V3.5.0Alpha 1 (21 Jul 2005)

What is it?

This hack provides a gateway from vBulletin forums to the USENET. It allows forum members to read newsgroup messages as well as to send posts to the newsgroups. In a nutshell, it turns vBulletin forums into a basic function news reader.

For an added bonus, this package also supports mailing list archive and forum to mailing list gateway. The software retrieves messages from a pop3 account and import them into the forums threaded. It also allows a forums user to send messages to the mailing list by the forum posting interface.

Why?

The gateway will be appreciated by forum members who do not know how to set up a news reader, or people who do not have port 119 open in their network. A forum interface to read and post news are often considered to be better than a news reader.

The gateway adds more contents to your forums and allows the members to interact with wider audience. It is common to see new users signing up to use this feature.

History

This hack was originally coded by Gilby for vB2. I modified the hack to work with vB3 and have been maintaining and improving the hack. I have now done the modification for this to work with vB3.5.

The very original hack by Gilby can be found here:

https://vborg.vbsupport.ru/showthrea...7&page=1&pp=15

The vB3 version can be found here:

https://vborg.vbsupport.ru/showthread.php?t=65152

Features

Some of the features and improvement include:

• Handles both newsgroup posts and mailing lists.
• Handles MIME encoded messages and attachment.
• Handles UUEncoded messages.
• Two Admin Control Panels for all settings.
• Automated installation.
• Correct thread implementation. All imported messages and posts sent to newsgroups will be threaded correctly according to References and Message-ID headers.
• Optional setting to thread by subject for mailing lists that strip references header.
• Imported messages will be searchable just like normal forum posts.
• Similar threads are built when the messages are imported if the system setting is on.
• Multiple attachments importing.
• Save attachment to database or file according to vB setting.
• Thumbnail created while importing image files.
• Multiple attachment post to newsgroup is also supported.
• Selectable X-No-Archive header honouring system.
• Light on system resources.
• Global Killfile to filter out unwanted messages. Killfiles can have OR and AND effectiveness.
• Strip footer by setting for incoming messages. Strips out repeated message footers.
• Built options to disable signatures and footers for out going messages.

Future development

• Traditional "> " lead posting interface.

What do you need and do not need?

You do not need to edit vB php files.
No need to manually run SQL.

Installation

Basically the AdminCP interfaces take care of installation as well as upgrade in the background. Back up your database first. The scripts can modify your database without alerting you!

Please follow the instructions in the readme.txt file for more details.

It is not a must to click "INSTALL" for this hack if you use it, I do try better answering the questions from people who do.

Support

Please post your support questions to this thread. Please check the FAQ in the second post within this thread first. It may be surprising, but it is possible your question is answered in the adminCP NNTP settings.

Please always provide more details about your problem. Examples and debug messages will certainly help. Well, if you don't even try to have some effort to compose your question, the answer you get may also not include much effort.

================== UPDATES ======================
Jul 21 2005 (release version 3.5.0 Alpha 1)

AdminCP xml file is written by Kadence.

• Ported to work with vB3.5.
• Existing bug fixes.
• Debug message can be sent by email or logged in vB Tast Scheduler.
• Known problem: image attachment thumb nail creation still needs work.
• Needs to be done: Documentation.

Quick installation instruction:
1) Upload files to their correct directory
2) Enter AdminCP, click on NNTP Settings once to install/upgrade. Change settings to your like.
3) Click on NNTP Groups once to install/upgrade.
3a) For new install. Run once
http://www.yourdomain.com/setlastmsg.php

Delete this file afterwards.
If you are upgrading from an older version. There is no need to upload this file. Actually this file should not be there except brand new install.

4)
Manually run in debug mode: http://forums.yourdomain.com/gateway.php?debug=1

[lierduh]

Originally Posted by tutorials.de

Hi together,

I've got a little tricky question for those NNTP Gateway experts out there

As you can see on the first attached screenshot (guests_on_forumdisplay.gif), the username is displayed in the "Last Post" column of forumdisplay.php. Those guys in the example are only "guest posters" from usenet but as you can see, everyone is displayed by his original username.

When we now head up to the next screenshot (guests_on_threaded_view.gif), we look inside an imported usenet thread, where the Threaded Mode feature of vBulletin is active. There you can see, that all guys are named "Newsgroup poster" (that's my alias for the usergroup guest).

I think it would be much better, if the naming of the authors in the Threaded / Hybrid mode box was the same as in forumdisplay.php - everyone, no matter if guest or registered member, should be displayed by his original, individual username. Does anyone have an idea how to achieve that?

Big thanks in advance & keep up the fabulous work!

- Dominik

The next version has the Ext User Prefix with all the messages imported. That would probably help people who are in your position wanting to distinguish internal posts and external posts.

You can see the samples:

http://phorums.com.au/forumdisplay.php?f=21

[lierduh]

Originally Posted by wcbryant

I've got what's probably a silly question, but after doing some reading and searches, haven't been able to find an answer or solution.

My forum defaults to linear, and 99% of my users stick with that. Does anyone know of a way where I can make a forum (the ones using this hack) default to threaded or hybrid mode for all users?

That is global/user setting, not by Forums. However it will be quite easy to implemented this as a hack through Plugin system. I will include this in the next version.

[wcbryant]

Originally Posted by lierduh

That is global/user setting, not by Forums. However it will be quite easy to implemented this as a hack through Plugin system. I will include this in the next version.

Wonderful. Thanks for the quick response.

I have this running on a test forum right now, sort of staging everything for my production move to 3.5.0. Anyway, after some feedback with my mods and admins, there's concern about the overwhelming volume of one of the newsgroups (alt.sport.football.college) we want to offer a gateway to.

Is there an option, or could there be an option, for imported threads/posts from offsite *not* to be counted towards the forum totals? So that forum post/thread totals are representative of local user contributions alone. For example, my little testing/staging forum is 2 days old, and has 80 or so local posts on it between myself, and my staff. But just one of the newsgroups has brought in over 7k posts in the same amount of time, thus skewing (severely) the impressions offered by post and thread totals throughout the site. Would be great if we had the ability to clean that up as described. 7k posts is a drop in the well when I move this over to my production side. But even over there, over the course of a year, this could add up very quickly.

Also, is a pruning option feasible? I seem to recall this came up elsewhere in the thread, but if it was, I just missed it browsing back through. I would love to see the option of making imported threads 'decay' after X days, and/or the option to limit the number of posts in one of the forums to Y, where everything older falls off. Either way, with the option (ideally) to make threads that local users have participated in exempt, for the sake of maintaining a relevant and meaningful history.

Just tossing these things out there. The extension is outstanding as-is, and my staff is ecstatic about it. Thanks so much for putting it together, and offering the support you do.

[PhillipH]

hm. anyone know if its possible to use google usenets? if so, the settings would be? thanks.

[Cyburbia]

Update - with Yahoo Groups, the gateway seems to be working flawlessly. The update from RC 1 to RC 2 didn't affect anything. I don't have a report on threading yet, since all the recent posts on the Yahoo group I'm gatewaying seem to be announcements.

A caveat: those using the extension should be aware that maintaining the subforum on the message board side takes a lot of work; there's cleaning up Yahoo-inserted message footers, out of office notifications, long legal disclaimers, and other text normally not found in message board posts. It's not too bad for slow lists, but for fast lists, it may take more moderation work than what would be encountered in a non-gatewayed group. For the gatewayed groups, I'm including this message below the page header:

To keep threads in this subforum readable, moderators may edit excessive backquoting, e-mail confidentiality notices and other legal disclaimers, and advertising inserted by the e-mail or mailing list server. They may also delete spam, out-of-office and similar notifications, and off-topic messages that are gatewayed from the mailing list.

See http://www.cyburbia.org/forums/forumdisplay.php?f=34

Thanks for a great script!

Originally Posted by PhillipH

hm. anyone know if its possible to use google usenets? if so, the settings would be? thanks.

I don't think Google is possible, unless there's a mail gateway of some sort. Maybe group alerts for Usenet --> vBulletin, but the other way around? I don't know.

You could try free NNTP servers -- there are many out there -- but odds are they'll be read-only. You probably won't be able to use your ISP's NNTP server because it'll be accessed from your Web host; most ISPs won't allow their NNTP servers to be accessed from an IP range outside what is assigned to that ISP.

If you really want to gateway with a Usenet group, your best bet is a cheap NNTP account with Giganews, EasyNews, or a similar Usenet provider, where you have full access to the NNTP server from any IP.

[eandhra]

hi, everybody,

I installed this mod, and working well.

can anybody please tell me the newsgroups/mailing list servers addresses, bcoz, i dont have any idea on this.

and give me suggestions, where and how to get these.

thanks

[frakman]

Okay, this is only a quick hack so I could import my MBOX file (extracted from Yahoo Goops) direct. (most because my server is running qmail).
I hope this gets included in standard. (does not require much polish)

First, install mbox.php from PEAR into the includes directory.

I don't think we are to post complete code - if I'm wong on that please tell me - so here are just the changes to pop.php:
Before

PHP Code:

class pop { 


PHP Code:

require_once('./includes/mbox.php'); 


After

PHP Code:

    function prepare_connection( $user, $pass, $host = "127.0.0.1", $port = "110" ) 


PHP Code:

        if (! (strpos($host, '/')===false) ) {    //mbox
            $this->mbox = true;        //mbox
            $this->_mbox = new Mail_Mbox();    //mbox
        } else {                    //mbox 


Replace

PHP Code:

        if ($host == '') { 
            return 0; 
        }
        $this->port = $port;
        $this->username = $user;
        $this->password = $pass;
        $this->host = $host; 


With

PHP Code:

            $this->password = $pass;
        }                    //mbox
        if ($host == '') {         //mbox
            return 0;             //mbox
        }                    //mbox
        $this->host = $host;        //mbox 


after

PHP Code:

    function connect()
    { 


Add

PHP Code:

//MBOX-START
        if ($this->mbox) {
            $this->pop_connect = $this->_mbox->open( $this->host );
            if (!$this->pop_connect) {
                echo "ERROR OPENING MBOX FILE\r\n";
                return -1;
            }


        } else {
//MBOX-END 


After

PHP Code:

        return -2; 


Add

PHP Code:

    }    //MBOX 


Replace this block

PHP Code:

    function howmany()
    {
        return $this->_howmany();
    } 


with

PHP Code:

    function _howmany()
    {
        if ($this->mbox) {  //MBOX
            return $this->_mbox->size( $this->pop_connect )-1;    //MBOX
        } else {            //MBOX
        $this->_write("STAT");
        $results = $this->_gets();
        $this->log .= $results;
        list ($results, $messages, $bytes) = split(' ', $results);
        return $messages;
        }                //MBOX
    } 


Lastly replace

PHP Code:

    function delete_mail ( $id )
    {
        $this->_write("DELE $id");
        $results = $this->_gets();
    } 


WITH

PHP Code:

    function delete_mail ( $id )
    {
        if ( !$this->mbox ) { //MBOX

        $this->_write("DELE $id");
        $results = $this->_gets();
        }                 //MBOX
    } 


When testing this I imported 10585 messages in 5m20s (not really a benchmark without doing it via POP3 as a comparison however surely the overheads are a lot less with this system)

How does this work? Put a file reference in the host box eg
/usr/local/apache2/www/mbox_locost_oz
where mbox_locost_oz is the MBOX archive. If you are on a Windows server the code would need to be changed (it looks for "/" which will not appear in a host name - would need to be changed to "\")

It will not delete messages from the archive. I do not see this as a bad thing for my purposes (ie dump in starting messages)

I also in my testing dug up a problem when there are no posts at all (I think its a NULL issue) I get this error:
Invalid SQL:

UPDATE vb_nntp_settings
		SET value =  WHERE varname = 'last_postid';

And this comes (in gateway.php)from

PHP Code:

$get_max_postid = $db->query_first("
    SELECT MAX(postid) as postid FROM " . TABLE_PREFIX . "post
"); 


ALSO somebody was complaining about lack of HTML formatting in Firefox:
in gateway.php change this line:
header("Content-Type: text/plain");
TO
header("Content-Type: text/html");

[Electronic Punk]

Having a problem that when I pull the threads in from the newsgroup it slows down my entire site because it locks the post table, does anyone else experience this?

the forum mysql logon does not have permission to lock tables, but seems to do it anyway.

[lierduh]

Originally Posted by Electronic Punk

Having a problem that when I pull the threads in from the newsgroup it slows down my entire site because it locks the post table, does anyone else experience this?

the forum mysql logon does not have permission to lock tables, but seems to do it anyway.

That is disadvantage of MYISAM table. Everytime you write to it. The entire table needs to be locked. INNODB on the other hand, does row level locking. The problem with INNODB is it does not offer Fulltext index (at least with the current version and near future versions). Also unlike MyISAM, InnoDB needs to do a full table scan for COUNT().

There is really nothing you can do about this. This is a on going performance problem faces anyone who owns a board which has many posts.

Perhaps you can put one second delay between post imports. Find this in the NNTP settings under AdminCP.

[Electronic Punk]

I have set a pause and will see how it goes.
Thanks.