NNTP Gateway for Usenet ( Newsgroups ), Mailing Lists

The latest version: V2.3.2 (31 Oct 2004)

What is it?

This hack provides a gateway from vBulletin forums to the USENET. It allows forum members to read newsgroup messages as well as to send posts to the newsgroups. In a nutshell, it turns vBulletin forums into a basic function news reader.

For an added bonus, this package also supports mailing list archive and forum to mailing list gateway. The software retrieves messages from a pop3 account and import them into the forums threaded. It also allows a forums user to send messages to the mailing list by the forum posting interface.

Why?

The gateway will be appreciated by forum members who do not know how to set up a news reader, or people who do not have port 119 open in their network. A forum interface to read and post news are often considered to be better than a news reader.

The gateway adds more contents to your forums and allows the members to interact with wider audience. It is common to see new users signing up to use this feature.

History

This hack is basically a hacked vB2 NNTP gateway written by Gilby to work with vB3. The original hack can be found here:

https://vborg.vbsupport.ru/showthrea...7&page=1&pp=15


Features

Some of the features and improvement include:

• Handles both newsgroup posts and mailing lists.
• Handles MIME encoded messages and attachment.
• Handles UUEncoded messages.
• Two Admin Control Panels for all settings.
• Automated installation.
• Correct thread implementation. All imported messages and posts sent to newsgroups will be threaded correctly according to References and Message-ID headers.
• Optional setting to thread by subject for mailing lists that strip references header.
• Supports TABLE_PREFIX.
• Imported messages will be searchable just like normal forum posts.
• Similar threads are built when the messages are imported if the system setting is on.
• Multiple attachments importing.
• Save attachment to database or file according to vB setting.
• Thumbnail created while importing image files.
• Multiple attachment post to newsgroup is also supported.
• Selectable X-No-Archive header honouring system.
• Light on system resources.
• Global Killfile to filter out unwanted messages. Killfiles can have OR and AND effectiveness.
• Strip footer by setting for incoming messages. Strips out repeated message footers.
• Built options to disable signatures and footers for out going messages.

Future development

• AdminCP interfaces. (completed)
• Installation/upgrade script. (completed, coded into AdminCP)
• Traditional "> " lead posting interface.

What do you need and do not need?

You do not need to edit vB php files if you use two bookmarks
You need to edit one vB php file to insert two lines of code if you want direct click links within AdminCP.
No need to run SQL.

Installation

Basically the AdminCP interfaces take care of installation as well as upgrade in the background. Back up your database first. The scripts can modify your database without alerting you!

Please follow the instructions in the readme.txt file for more details.

Support

Please post your support questions to this thread. Please check the FAQ in the third post within this thread first.

================== UPDATES ======================
Jul 31 2004 (release version 2.2.0)

• Provided option of not importing attachments
• Provided option of pause between messages
• Provided option of not sending messages to USENET
• Provided mechanism to skip importing a message if it was crashed on it
• Provided "Default Footer" option
• Provided strip_footer function for mailing lists
  (written originally by KevinM)
• Provided Killfile functions with Header/Body/AND/OR settings
• Provided ability to thread messages by subject for mailing lists
• Provided ability to use a single POP3 account for different mailing lists
  (use prefix)
• Merged KevinM's attachment routine, save thumbnail to hard drive
  as vB3.0.3
• Official support to have the script run by Scheduled Tasks
• Many bug fixes

Please note, when you manually run the script, you must turn on the debug mode. Otherwise nothing will be displayed on the screen.

To run as debug mode: http://forums.yourdomain.com/gateway.php?debug=1

25 Oct 2004

• Provided the option of sending posts using the real email address
• Provided the option of sending posts using a fixed email address for
  each group.
• Posts are now held back if they are in the moderating queue
• Scheduled task log is now working
• Provided the option of sending the debug messages to the vB3 cron log
• Bug fixes

31 Oct 2004

Fixed bugs running the gateway by the vB3 Scheduled Tasks

1 March 2004

Uploaded the new zip file. It contains one or two bug fixes.

[AWS]

Originally Posted by Kadence

Ah, duh?$debug is used when passing ?debug=1 in from the URL Without it the logging() function doesn't work.

So I added the following lines after line 71 in gateway.php:

PHP Code:

$log = $vbulletin->GPC['log'];
$debug = $vbulletin->GPC['debug']; 


The .zip file in the post above has been updated.

Thanks much. I'll give it a try on my test box.

[Gottcha]

can someone help me here.... i have most of it all installed but the file tries to download...so i went to the ms site and attempted to edit the registry. i found the key and added a new value and got it all up to the point of adding the value data. It wont let me add HEX 0x1. Id really like to install this and get it all working...thanks in advance.

Key: HKEY_CURRENT_USER\Software\Microsoft\Windows\Curre ntVersion\Internet Settings

Value name: IsTextPlainHonored
Value type: DWORD
Value data: HEX 0x1

FIXED IT....WELL ACTUALLY RAN THE FILE ON ANOTHER MACHINE

[Kadence]

OK?I made some more changes

There was an issue with form variable names on nntp_groups.php line 278 (original line 256), of the same variety as the print_forum_chooser problem on line 263 (241). This was causing problems with "Add Newsgroup", rather than "Edit"; when trying to add a newsgroup, the forum would be set to 0, rather than the forumid.

Also, I replaced all instances of $vboptions['...'] with $vbulletin->options['...'] and global $vboptions with global $vbulletin in functions_nntp.php. Looks like this affects a lot of attachment stuff.

I have no idea if attachments work now; perhaps someone will test it and share their results :squareeyed:

My earlier post has the updated .zip file.

[danrak]

Would it be possible to use this to share posts between sites? How would it be done?

[lierduh]

Originally Posted by Kadence

Tremendous mod

Only problem is the speed. I seem to be getting only about 3,500 posts/hour, whereas it takes under a minute to download that number through NNTP on a newsreader. But I suppose that's probably because of the vBulletin database functions taking up a lot of time, rather than the download of articles itself...?

vB indexes all the words in the post contents, that is why it is very fast to search a term from the post. Whereas a normal news reader usually only index the subject. To find a keyword within the post would basically require a full article scan. I believe the indexing takes up most of the time for importing.

To speed up a bit, one can delete the index for 'msgid' in the post table, and reindex that field with a smaller size. It is quite easy to do with myphpadmin. Just set the size of the index to 10 or 15. That might(according to mysql manual) improve some speed.

[lierduh]

Originally Posted by Skalek

Yeah I got that all working, but now I have a problem where the script is not following the its own time constraints. From what I understand it is supposed to run no longer than 30 minutes. Also another instance should not run within 30 minutes of the first starting. This makes sense so that you dont have multiple copies running at the same time.

My problem is that gateway.php seems to be running for 2 hours + so that if I cron it multiple instances will occur. Anyone else run into this problem?

I actually mentioned this somewhere, if not in the instruction, it will be somewhere in this thread.

The script tries to check if there is another instance of script running within 30 minutes from the time the last script was run. It will ignore the checking if it is over 30 minutes and considers the last run crashed.

So it is important not to set up the cron job before the completion of the initial import, as the initial importation may take too much time.

[krohnathlonman]

Originally Posted by lierduh

vB indexes all the words in the post contents, that is why it is very fast to search a term from the post. Whereas a normal news reader usually only index the subject. To find a keyword within the post would basically require a full article scan. I believe the indexing takes up most of the time for importing.

To speed up a bit, one can delete the index for 'msgid' in the post table, and reindex that field with a smaller size. It is quite easy to do with myphpadmin. Just set the size of the index to 10 or 15. That might(according to mysql manual) improve some speed.

Could you give us some more information on this as well as anything else that can speed the import process up... So far I've just been tossing SCSI drives at it and simply waiting LOL

[lierduh]

Originally Posted by krohnathlonman

Could you give us some more information on this as well as anything else that can speed the import process up... So far I've just been tossing SCSI drives at it and simply waiting LOL

To delete the old index:

ALTER TABLE `post` DROP INDEX `msgid`;

and to add a partial index for msgid:

ALTER TABLE `post` ADD INDEX ( `msgid` ( 10 ) ) ;

10 means instead of indexing the whole length of the field (128), we only index the first 10 charactors. This will reduce the size of the index. The full length index is fastest at the time of retrieving, a partial index means the DB engine needs to read more lines at the time of retrieving but needs less work at writing.

To people who needs to import large amount of posts in one go. I think if you disable the index building process at the time of importing, it will speed up a lot. You can then rebuild the index using the AdminCP. Bear in mind, rebuilding the index takes a VERY long time. The total time used to rebuild the index should be less than the time used to import large amount of messages and build the index at the same time. I have not done this, this is only base on the theory that it is very slow to insert records into a fulltext index table, and it is faster to build the fulltext index after all the insertions are done.

If you do not need to search the imported messages, then disabling the index build will surely speed up the import. To do this, just comment out

build_post_index($postid, $foruminfo);

line in the /include/functions_nntp.php

[vprp]

I'm gonna go ahead and test the 3.5 version on my test version. I'll let you all know how things go.

[krohnathlonman]

the 3.5 version works good an does support attachments from the testing I've done....