NNTP Gateway for Usenet ( Newsgroups ), Mailing Lists

The latest version: V3.5.0Alpha 1 (21 Jul 2005)

What is it?

This hack provides a gateway from vBulletin forums to the USENET. It allows forum members to read newsgroup messages as well as to send posts to the newsgroups. In a nutshell, it turns vBulletin forums into a basic function news reader.

For an added bonus, this package also supports mailing list archive and forum to mailing list gateway. The software retrieves messages from a pop3 account and import them into the forums threaded. It also allows a forums user to send messages to the mailing list by the forum posting interface.

Why?

The gateway will be appreciated by forum members who do not know how to set up a news reader, or people who do not have port 119 open in their network. A forum interface to read and post news are often considered to be better than a news reader.

The gateway adds more contents to your forums and allows the members to interact with wider audience. It is common to see new users signing up to use this feature.

History

This hack was originally coded by Gilby for vB2. I modified the hack to work with vB3 and have been maintaining and improving the hack. I have now done the modification for this to work with vB3.5.

The very original hack by Gilby can be found here:

https://vborg.vbsupport.ru/showthrea...7&page=1&pp=15

The vB3 version can be found here:

https://vborg.vbsupport.ru/showthread.php?t=65152

Features

Some of the features and improvement include:

• Handles both newsgroup posts and mailing lists.
• Handles MIME encoded messages and attachment.
• Handles UUEncoded messages.
• Two Admin Control Panels for all settings.
• Automated installation.
• Correct thread implementation. All imported messages and posts sent to newsgroups will be threaded correctly according to References and Message-ID headers.
• Optional setting to thread by subject for mailing lists that strip references header.
• Imported messages will be searchable just like normal forum posts.
• Similar threads are built when the messages are imported if the system setting is on.
• Multiple attachments importing.
• Save attachment to database or file according to vB setting.
• Thumbnail created while importing image files.
• Multiple attachment post to newsgroup is also supported.
• Selectable X-No-Archive header honouring system.
• Light on system resources.
• Global Killfile to filter out unwanted messages. Killfiles can have OR and AND effectiveness.
• Strip footer by setting for incoming messages. Strips out repeated message footers.
• Built options to disable signatures and footers for out going messages.

Future development

• Traditional "> " lead posting interface.

What do you need and do not need?

You do not need to edit vB php files.
No need to manually run SQL.

Installation

Basically the AdminCP interfaces take care of installation as well as upgrade in the background. Back up your database first. The scripts can modify your database without alerting you!

Please follow the instructions in the readme.txt file for more details.

It is not a must to click "INSTALL" for this hack if you use it, I do try better answering the questions from people who do.

Support

Please post your support questions to this thread. Please check the FAQ in the second post within this thread first. It may be surprising, but it is possible your question is answered in the adminCP NNTP settings.

Please always provide more details about your problem. Examples and debug messages will certainly help. Well, if you don't even try to have some effort to compose your question, the answer you get may also not include much effort.

================== UPDATES ======================
Jul 21 2005 (release version 3.5.0 Alpha 1)

AdminCP xml file is written by Kadence.

• Ported to work with vB3.5.
• Existing bug fixes.
• Debug message can be sent by email or logged in vB Tast Scheduler.
• Known problem: image attachment thumb nail creation still needs work.
• Needs to be done: Documentation.

Quick installation instruction:
1) Upload files to their correct directory
2) Enter AdminCP, click on NNTP Settings once to install/upgrade. Change settings to your like.
3) Click on NNTP Groups once to install/upgrade.
3a) For new install. Run once
http://www.yourdomain.com/setlastmsg.php

Delete this file afterwards.
If you are upgrading from an older version. There is no need to upload this file. Actually this file should not be there except brand new install.

4)
Manually run in debug mode: http://forums.yourdomain.com/gateway.php?debug=1

[toibs]

thats strange - i didnt have that problem at all... luckilly enough it worked "straight out the box" (Tho still have the above mentioned problems with email.... )

[dkendall]

Originally Posted by toibs

thats strange - i didnt have that problem at all...

It may only show up if the first post in the newsgroup has index 1. Since old posts can get archived/removed, it's possible your newgroups started with a post higher than 1.

Also, the problem (mentioned long ago) with the gateway spewing HTML when used with FireFox and the scheduled task manager is caused by this:

if (!headers_sent())
{
	header("Content-Type: text/plain");
}

... around line 35 of gateway.php. Just take (or comment) it out, and the problem goes away.

David

[dkendall]

Another issue I've run into...

If an NNTP user has not provided a display name (at least in Outlook Express), the "From" field in his posts is "<name@domain.tld>" and the from_name function in functions_nntp.php returns an empty string.

My workaround is to add this:

if(!$from_name)
    $from_name = from_email($from_raw);

Just before this:

return $from_name;

Then I made the email obfusication mods described here:

https://vborg.vbsupport.ru/showpost....&postcount=209

... so that only the "name" portion of the e-mail address is displayed.

Hopefully, the maintainer of this code can add this to the official version

David

[dkendall]

Originally Posted by dkendall

Also, the problem (mentioned long ago) with the gateway spewing HTML when used with FireFox and the scheduled task manager is caused by this:

if (!headers_sent())
{
	header("Content-Type: text/plain");
}

... around line 35 of gateway.php. Just take (or comment) it out, and the problem goes away.

Another thing... If you want to be able to run the gateway in debug mode via the Scheduled Task manager, replace function logging in functions_nntp.php with this:

function logging($text)
{
	global $nntp;

	if (isset($nntp['debug']) AND $nntp['debug'] == 1)
	{
		if (function_exists('log_cron_action'))
		{
			print_r($text . "<br>\r\n");
		}
		else
		{
			if (!headers_sent())
			{
				header("Content-Type: text/plain");
			}
			echo $text . "\r\n";
			flush();
			ob_flush();
		}
	}

	if ($nntp['settings']['do_cron_log'])
	{
		$GLOBALS['nntp']['cron_log'] .= $text . "<br />";
	}
}

This will stop it spewing raw HTML.

AFAIK, the scheduler won't allow you do specify the DEBUG parameter, so you'll also have to modify gateway.php and add this:

$nntp['debug'] = 1;

David

[dkendall]

I found and fixed a few bugs:

1. When an article is imported to a sub-forum, the NNTP gateway was doubling thread and post counts in the parent forum(s).
   
2. It was updating the "latest post" fields, even if the forum (or parent forum) has newer posts.
   
3. The postings added by the gateway were not inheriting the icon of the thread to which they were added (if I understand this concept correctly).

I wanted imported NNTP messages to be linked to the vBulletin user profile of the poster. As the code is intertwined with the above bug fixes, I've included it here too.

To get the vBulletin user id from the NNTP post e-mail address, locate the following lines in gateway.php:

	//Separate name and email address
	$from_name = from_name($message['from']);
	$from_email = from_email($message['from']);

Add the following lines immediately beneath:

	//Get user id from email address
	$from_userid = 0;
	if ($from_email AND ($nntp_settings['associate_by_email'] == 1))
	{
		$user_data = $db->query_first("SELECT userid FROM "
			. TABLE_PREFIX . "user WHERE email='" . addslashes($from_email) . "'");
		if (!empty($user_data))
			$from_userid = $user_data['userid'];
	}

Note that we require each unique email addresses in vBulletin.

Fix up the SQL statements in the if ($threadid) / else block in gateway.php as follows:

	// if the correct thread was found, insert it.
	if ($threadid) {
		$lasticonid = 0;
		$icon_data = $db->query_first("SELECT iconid FROM " . TABLE_PREFIX . "thread WHERE threadid=$threadid");
		if (!empty($icon_data))
			$lasticonid = $icon_data['iconid'];

		$postid = insert_post($threadid, $forumid, $foruminfo, $subject, $from_name, $from_email, $from_userid, $date, $lasticonid, $parentid);

		// update thread
		$db->query("UPDATE " . TABLE_PREFIX . "thread
			SET lastpost = '" . $date . "',
			replycount = replycount + 1,
			lastposter = '" . addslashes($from_name) . "'
			WHERE threadid = $threadid
		");

		// update the forum counts
		$db->query("UPDATE " . TABLE_PREFIX . "forum
			SET replycount = replycount + 1
			WHERE forumid = $forumid");

		// update forum last-post data
		$db->query("
			UPDATE " . TABLE_PREFIX . "forum
			SET lastpost = '" . $date . "',
			lastposter = '" . addslashes($from_name) . "',
			lasticonid = $lasticonid,
			lastthreadid = $threadid,
			lastthread = '" . addslashes($subject) . "'
			WHERE forumid IN ({$foruminfo['parentlist']})
			AND lastpost < '" . $date . "'
		");


		// send out email notices
		exec_send_notification($threadid, "0", $postid);
	} else {
		//can only be here if no thread is found
		//Needs to create new thread

		// Create thread
		if ($vbulletin->options['similarthreadsearch'])
		{
			require_once(MY_DIR . '/includes/functions_search.php');
			$similarthreads = fetch_similar_threads($subject);
		}
		else
		{
			$similarthreads = '';
		}

		$db->query("INSERT INTO " . TABLE_PREFIX . "thread 
			(title, lastpost, forumid, open, replycount,
			postusername, postuserid, lastposter, dateline, iconid,
			visible, views, similar)
			VALUES ('" . addslashes($subject) . "', '" . $date . "', $forumid, 1, 0,
			'" . addslashes($from_name) . "', $from_userid,
			'" . addslashes($from_name) . "', '" . $date . "', 0, 1, 0,
			'" . $similarthreads . "')
		");


		$threadid = $db->insert_id();

		//insert_post
		$postid = insert_post($threadid, $forumid, $foruminfo, $subject, $from_name, $from_email, $from_userid, $date);

		// update the forum counts
		$db->query("UPDATE " . TABLE_PREFIX . "forum
			SET replycount = replycount + 1,
			threadcount = threadcount + 1
			WHERE forumid = $forumid");

		// update forum last-post data
		$db->query("UPDATE " . TABLE_PREFIX . "forum
			SET lastpost = '" . $date . "',
			lastposter = '" . addslashes($from_name) . "',
			lasticonid = 0,
			lastthread = '" . addslashes($subject) . "',
			lastthreadid = $threadid
			WHERE forumid IN ({$foruminfo['parentlist']})
			AND lastpost < '" . $date . "'
			");

		logging("'$subject' from ". $from_name . ". New thread.");

	} //new thread or not

In functions_nntp.php replace function insert_post with the following:

//Insert post and follow up

function insert_post($threadid, $forumid, $foruminfo, $subject, $from_name, $from_email, $from_userid, $date, $iconid = 0, $parentid = 0)
{
	global $db, $nntp;

	$message =& $nntp['message'];

	$db->query("INSERT INTO " . TABLE_PREFIX . "post
		(postid, threadid, title, username, userid, dateline, pagetext,
		allowsmilie, showsignature, ipaddress, iconid, visible,
		isusenetpost, msgid, ref, parentid) VALUES
		(NULL, $threadid, '". addslashes($subject) . "',
		'" . addslashes($from_name) . "', $from_userid, '" . $date . "',
		'" . addslashes($message['text']) . "', 1, 0,
		'" . addslashes($from_email) . "', $iconid, 1, 1,
		'" . addslashes($message['message-id']) . "',
		'" . addslashes($message['references']) . "', "
		. $parentid . ")");

	$postid=$db->insert_id();

	//So that thread preview works
        $db->query("
		UPDATE " . TABLE_PREFIX . "thread
                SET firstpostid = $postid
		WHERE threadid = $threadid
	");

	// ULTRASOFT HACK
	if($from_userid != 0)
	{
		// update user's post count
		$db->query("UPDATE " . TABLE_PREFIX . "user
			SET lastpost = '" . $date . "',
			lastactivity = '" . $date . "',
			posts = posts + 1
			WHERE userid = $from_userid
		");
	}

	//save attachments if any
	if ($message['attachments'])
	{
		process_attachments($date, $postid, $threadid, $forumid);
	}

	// Index post for searching
	build_post_index($postid, $foruminfo);

	return $postid;
}

In AdminCP go the to NNTP Gateway Settings, click the Add a New Setting button, and add this:

	Title: Associate by Email
	Varname: associate_by_email
	Value: 1
	Description: Use the email address to associate postings with the corresponding vBulletin user.

Hopefully, the owner/maintainer of this hack can add this to the "official" version.

David

[dkendall]

Here's another mod I've made to this hack.

In wrestling with getting it working, I needed to blow away my vBulletin threads/posts several times, and re-import everything from USENET. Since some of the postings originated in vBulletin, it was skipping them (which was mucking up the threading).

Locate the whole if (trim($message['user-agent'])... block of code in gateway.php and tweak it as follows:

	$skip_post = 0;

	if (trim($message['user-agent']) == trim($nntp_settings['useragent'])
		AND $nntp_settings['organization_check'] == 0)
	{
		if ($nntp_settings['full_import_from_usenet'] == 0) {
			logging("Skip, post was sent from our forum.");
			$skip_post = 1;
		}
	}
	else if (trim($message['user-agent']) == trim($nntp_settings['useragent'])
		AND $nntp_settings['organization_check'] == 1
		AND trim($message['organization']) == trim($nntp_settings['organization']))
	{
		//added organization so that we don't skip fellow gateway's posts
		if ($nntp_settings['full_import_from_usenet'] == 0) {
			logging("Skip, post was sent from our forum.");
			$skip_post = 1;
		}
	}
	else if (stristr(trim($message['x-no-archive']), 'yes') AND
		isset($nntp_settings['honor_no-archive']) AND
		$nntp_settings['honor_no-archive'] != 0)
	{
		logging("Skip, X-No-Archive headers is set.");
	}
	elseif ($nntp['grouptype'] == 'mail'
		AND $group['prefix']
		AND stristr($message['subject'], $group['prefix']) == false)
	{
		logging("Skip, not matching prefix: \"" . $group['prefix'] . "\"");
	}

	if(!$skip_post)
	{
	   $kf = killfile_match();

In AdminCP go the to NNTP Gateway Settings, click the Add a New Setting button, and add this:

	Title: Full import from USENET
	Varname: full_import_from_usenet
	Value: 0
	Description: Override useragent and organization checks to import absolutely everything from USENET.

In order to use this feature, you have to:

1. Enable it by setting the full_import_from_usenet value to 1.
2. Delete everything from the thread and post tables in the vBulletin database (selectively, if you want to re-import only certain newsgroups or posts).
3. In AdminCP go to NNTP Gateway Newsgroups and set the Last message number to zero (or some appropriate value).
4. Run Update Counters and rebuild the thread and forum information, and Update Post Counts.
5. Run gateway.php script.
6. Reset the full_import_from_usenet value to zero.

David

[dkendall]

I found that the NNTP gateway was choking on some articles that had a period on a line by itself.

This seems to be used to indicate the end of the article, but in fact, there was more text to follow.

I worked around this by changing function get_article in nntp.php. I replaced this code:

	while(!feof($this->fp)) {
		//$line = trim(fgets($this->fp, 256));
		//Took out the length, some lines are longer than that
	    $line = trim(fgets($this->fp));

	    if ($line == ".") {
		break;
	    } else {
		$post .= $line ."\n";
	    }
	}

...with this:

	while(!feof($this->fp)) {
	    $line = fgets($this->fp);

	    if ($line == ".\r\n") {
		break;
	    } else {
		$post .= rtrim($line, "\r\n") ."\n";
	    }
	}

BTW, the original code appears to strip leading blanks from each line. This is probably wrong, so I made it strip only the cr/lf.

David

[Deepdog009]

Hey dkendall glad 2 C U be on the Ball with yo add ons. This hack has some po 10 show if it be enhanced and bug free.

I had a problem with it last month! Some groups dont like link backs to userid and forumid. I X-nessed all dat stuff ( signiture, id, links and anything dat might get da boot ) from my ex-posts. Usenet groups be picky when importing from the groups and exporting. All newbies trying diss out be careful when posting to Usenet. Dont put more than one group per forum!!! Mod yo posts.

Got a ???
Can U fix it so that can delete posts from a certain Usenet user in VBull??? When I try to delete a user from Usenet in VBull I get message user not found goback. Got posts by user but VBull says cant find user, what the hec.

Can U tell me how to fix???

[Grinler]

Wow...that took me about 10 reads to fully understand you. Usenet post are not posted as real vbulletin users..they are actually considered guest posts.

[dkendall]

Originally Posted by Deepdog009

Some groups dont like link backs to userid and forumid. I X-nessed all dat stuff ( signiture, id, links and anything dat might get da boot ) from my ex-posts.

I'm not sure what this means (I'm a vBulleting newbie myself). How do you put links to userid, forum id, etc., into a post? And how do they look when they're exported to the usenet group?

Originally Posted by Deepdog009

Can U fix it so that can delete posts from a certain Usenet user in VBull??? When I try to delete a user from Usenet in VBull I get message user not found goback.

With the patch I posted previously, imported usenet posts will be linked to VB profiles if their email addresses are found in VB. Otherwise, they're guest posts.

I suppose it would be possible to automatically register unknown email addresses in VB (perhaps using the email address as the username too), but I don't think that's a feature I would want to use.

David