NNTP Gateway for Usenet ( Newsgroups ), Mailing Lists

The latest version: V3.5.0Alpha 1 (21 Jul 2005)

What is it?

This hack provides a gateway from vBulletin forums to the USENET. It allows forum members to read newsgroup messages as well as to send posts to the newsgroups. In a nutshell, it turns vBulletin forums into a basic function news reader.

For an added bonus, this package also supports mailing list archive and forum to mailing list gateway. The software retrieves messages from a pop3 account and import them into the forums threaded. It also allows a forums user to send messages to the mailing list by the forum posting interface.

Why?

The gateway will be appreciated by forum members who do not know how to set up a news reader, or people who do not have port 119 open in their network. A forum interface to read and post news are often considered to be better than a news reader.

The gateway adds more contents to your forums and allows the members to interact with wider audience. It is common to see new users signing up to use this feature.

History

This hack was originally coded by Gilby for vB2. I modified the hack to work with vB3 and have been maintaining and improving the hack. I have now done the modification for this to work with vB3.5.

The very original hack by Gilby can be found here:

https://vborg.vbsupport.ru/showthrea...7&page=1&pp=15

The vB3 version can be found here:

https://vborg.vbsupport.ru/showthread.php?t=65152

Features

Some of the features and improvement include:

• Handles both newsgroup posts and mailing lists.
• Handles MIME encoded messages and attachment.
• Handles UUEncoded messages.
• Two Admin Control Panels for all settings.
• Automated installation.
• Correct thread implementation. All imported messages and posts sent to newsgroups will be threaded correctly according to References and Message-ID headers.
• Optional setting to thread by subject for mailing lists that strip references header.
• Imported messages will be searchable just like normal forum posts.
• Similar threads are built when the messages are imported if the system setting is on.
• Multiple attachments importing.
• Save attachment to database or file according to vB setting.
• Thumbnail created while importing image files.
• Multiple attachment post to newsgroup is also supported.
• Selectable X-No-Archive header honouring system.
• Light on system resources.
• Global Killfile to filter out unwanted messages. Killfiles can have OR and AND effectiveness.
• Strip footer by setting for incoming messages. Strips out repeated message footers.
• Built options to disable signatures and footers for out going messages.

Future development

• Traditional "> " lead posting interface.

What do you need and do not need?

You do not need to edit vB php files.
No need to manually run SQL.

Installation

Basically the AdminCP interfaces take care of installation as well as upgrade in the background. Back up your database first. The scripts can modify your database without alerting you!

Please follow the instructions in the readme.txt file for more details.

It is not a must to click "INSTALL" for this hack if you use it, I do try better answering the questions from people who do.

Support

Please post your support questions to this thread. Please check the FAQ in the second post within this thread first. It may be surprising, but it is possible your question is answered in the adminCP NNTP settings.

Please always provide more details about your problem. Examples and debug messages will certainly help. Well, if you don't even try to have some effort to compose your question, the answer you get may also not include much effort.

================== UPDATES ======================
Jul 21 2005 (release version 3.5.0 Alpha 1)

AdminCP xml file is written by Kadence.

• Ported to work with vB3.5.
• Existing bug fixes.
• Debug message can be sent by email or logged in vB Tast Scheduler.
• Known problem: image attachment thumb nail creation still needs work.
• Needs to be done: Documentation.

Quick installation instruction:
1) Upload files to their correct directory
2) Enter AdminCP, click on NNTP Settings once to install/upgrade. Change settings to your like.
3) Click on NNTP Groups once to install/upgrade.
3a) For new install. Run once
http://www.yourdomain.com/setlastmsg.php

Delete this file afterwards.
If you are upgrading from an older version. There is no need to upload this file. Actually this file should not be there except brand new install.

4)
Manually run in debug mode: http://forums.yourdomain.com/gateway.php?debug=1

[lierduh]

Originally Posted by chrispadfield

This code change (an ugly hack, but i was trying to be quick) fixed it:

	if ($get_max_postid['postid'] < 1) {
		$get_max_postid['postid'] = 0;
	}
	$db->query("UPDATE " . TABLE_PREFIX . "nntp_settings
		SET value = " . $get_max_postid['postid'] .
		" WHERE varname = 'last_postid'
	");

The problem happens when you have a completely empty forum. I fixed that in the CVS a while back, but having trouble to find time to release a new version.

[Krisekocm]

When i run gateway.php the script is importing only from one group !
but i setup up 2 groups how to make this working to update both ?

[oezer]

Originally Posted by lierduh

There is your anwser.

hi lierduh,

i have disabled the posting function on the settings, but the scripts imports only 3 threads and breaks. i got no errors. have you any ideas why the script is not normally finished at the end of the time?

Gateway version 2.3.2 1 group(s) gatewayed.

Connecting to server, server says: 200 server ready - posting allowed

Info for adobe.acrobat.macintosh at adobeforums.com: 211 24126 21534 45659 adobe.acrobat.macintosh group selected

Getting message number 45295:
'Re: Calculation Forms' from Anne_Dewey@adobeforums.com. Thread found by References.
Getting message number 45296:
'Re: Slow Performance of Acrobat Form' from Anne_Dewey@adobeforums.com. Thread found by References.
Getting message number 45297:

on the 3rd message the script hangs up

[scruby]

Originally Posted by lierduh

That is global/user setting, not by Forums. However it will be quite easy to implemented this as a hack through Plugin system. I will include this in the next version.

I got tired of waiting, so I threw together a quickie plugin that does this.

1. add a plugin, the hook you want is "showthread_start"
2. product is vbulletin (or you can create a product just for nntp hacks)
3. PHP Code:

   global $vbulletin;
$thread = verify_id('thread', $threadid, 1, 1);
if ($thread) {    
    $sql = "SELECT newsgroupid FROM " . 
       TABLE_PREFIX . 
       "nntp_groups WHERE forum = " .
       $thread['forumid'];
    if($vbulletin->db->query_first($sql)) {
        unset($threadedmode);
        $vbulletin->userinfo['threadedmode'] = 2; // 1 - threaded, 2 - hybrid
    }
} 


This makes it so that usenet threads are forced into beng threaded or hybrid mode (you pick which)

[scruby]

This has come up before in the threads for the earlier versions of this hack, but I'm bringing it up again since this version of the nntp gatway is still in the alpha phase of development.

In a moderated newsgroup, it's disruptive to post replies to articles that weren't approved by the newsgroup's moderators.

I want to find a good way to deal with moderated newsgroups so that articles and threads don't show up immediately.

Ideally, outgoing messages would be queued for gateway.php in tables outside of vbulletin's thread and post tables, and wouldn't appear in the forums until the message is downloaded from the news server.

[dethfire]

lierduh, thanks for your hard work, any idea when the next version will be released?

[moftary]

Originally Posted by frakman

Okay, this is only a quick hack so I could import my MBOX file (extracted from Yahoo Goops) direct. (most because my server is running qmail).
I hope this gets included in standard. (does not require much polish)

First, install mbox.php from PEAR into the includes directory.

I don't think we are to post complete code - if I'm wong on that please tell me - so here are just the changes to pop.php:
Before

PHP Code:

class pop { 


PHP Code:

require_once('./includes/mbox.php'); 


After

PHP Code:

    function prepare_connection( $user, $pass, $host = "127.0.0.1", $port = "110" ) 


PHP Code:

        if (! (strpos($host, '/')===false) ) {    //mbox
            $this->mbox = true;        //mbox
            $this->_mbox = new Mail_Mbox();    //mbox
        } else {                    //mbox 


Replace

PHP Code:

        if ($host == '') { 
            return 0; 
        }
        $this->port = $port;
        $this->username = $user;
        $this->password = $pass;
        $this->host = $host; 


With

PHP Code:

            $this->password = $pass;
        }                    //mbox
        if ($host == '') {         //mbox
            return 0;             //mbox
        }                    //mbox
        $this->host = $host;        //mbox 


after

PHP Code:

    function connect()
    { 


Add

PHP Code:

//MBOX-START
        if ($this->mbox) {
            $this->pop_connect = $this->_mbox->open( $this->host );
            if (!$this->pop_connect) {
                echo "ERROR OPENING MBOX FILE\r\n";
                return -1;
            }


        } else {
//MBOX-END 


After

PHP Code:

        return -2; 


Add

PHP Code:

    }    //MBOX 


Replace this block

PHP Code:

    function howmany()
    {
        return $this->_howmany();
    } 


with

PHP Code:

    function _howmany()
    {
        if ($this->mbox) {  //MBOX
            return $this->_mbox->size( $this->pop_connect )-1;    //MBOX
        } else {            //MBOX
        $this->_write("STAT");
        $results = $this->_gets();
        $this->log .= $results;
        list ($results, $messages, $bytes) = split(' ', $results);
        return $messages;
        }                //MBOX
    } 


Lastly replace

PHP Code:

    function delete_mail ( $id )
    {
        $this->_write("DELE $id");
        $results = $this->_gets();
    } 


WITH

PHP Code:

    function delete_mail ( $id )
    {
        if ( !$this->mbox ) { //MBOX

        $this->_write("DELE $id");
        $results = $this->_gets();
        }                 //MBOX
    } 


When testing this I imported 10585 messages in 5m20s (not really a benchmark without doing it via POP3 as a comparison however surely the overheads are a lot less with this system)

How does this work? Put a file reference in the host box eg
/usr/local/apache2/www/mbox_locost_oz
where mbox_locost_oz is the MBOX archive. If you are on a Windows server the code would need to be changed (it looks for "/" which will not appear in a host name - would need to be changed to "\")

It will not delete messages from the archive. I do not see this as a bad thing for my purposes (ie dump in starting messages)

I also in my testing dug up a problem when there are no posts at all (I think its a NULL issue) I get this error:
Invalid SQL:

UPDATE vb_nntp_settings
		SET value =  WHERE varname = 'last_postid';

And this comes (in gateway.php)from

PHP Code:

$get_max_postid = $db->query_first("
    SELECT MAX(postid) as postid FROM " . TABLE_PREFIX . "post
"); 


ALSO somebody was complaining about lack of HTML formatting in Firefox:
in gateway.php change this line:
header("Content-Type: text/plain");
TO
header("Content-Type: text/html");

Or you could simply use /var/qmail/bin/maildir2mbox

[moftary]

Originally Posted by donnacha

Oh c'mon, that's just standard contract copperplate, doesn't mean anything other than they'd prefer you not to run your own ISP off any of their accounts. With an account of this kind, downloading miniscule amounts of text-only posts, you are, literally, the least of their worries.

For God's sake, we're talking about $10!!! If, in a sudden fit of insanity, Astraweb decide to mobilise their legal department, go elsewhere.

Nuh hunney
I had my giganews account phrozed for that. When your forum get popular, a few of your users will spam newsgroups out of your forum getting your account to be suspended.

[moftary]

Feature Request:

- Ignore messages with no reference 1/0

[KeKe]

I'm trying to install in ASmallOrange hosting server.

When I run gateway.php?debug=1

All I get is:
*****
Gateway version 2.3.2 1 group(s) gatewayed.

Connecting to server, server says: 200 NNTP Service 6.0.3790.206 Version: 6.0.3790.206 Posting Allowed

Info for microsoft.public.windowsxp.general at msnews.microsoft.com: 211 53164 1 1442385 microsoft.public.windowsxp.general
*****

I have tried waiting for 10 minutes, nothing happens.

Should I ask my host to open any port or something?

I have tried in 3.0.7 with 2.3.2 AND 3.5.2 with 3.5. It gets stuck in the same gateway thing.