NNTP Gateway for Usenet ( Newsgroups ), Mailing Lists

The latest version: V2.3.2 (31 Oct 2004)

What is it?

This hack provides a gateway from vBulletin forums to the USENET. It allows forum members to read newsgroup messages as well as to send posts to the newsgroups. In a nutshell, it turns vBulletin forums into a basic function news reader.

For an added bonus, this package also supports mailing list archive and forum to mailing list gateway. The software retrieves messages from a pop3 account and import them into the forums threaded. It also allows a forums user to send messages to the mailing list by the forum posting interface.

Why?

The gateway will be appreciated by forum members who do not know how to set up a news reader, or people who do not have port 119 open in their network. A forum interface to read and post news are often considered to be better than a news reader.

The gateway adds more contents to your forums and allows the members to interact with wider audience. It is common to see new users signing up to use this feature.

History

This hack is basically a hacked vB2 NNTP gateway written by Gilby to work with vB3. The original hack can be found here:

https://vborg.vbsupport.ru/showthrea...7&page=1&pp=15


Features

Some of the features and improvement include:

• Handles both newsgroup posts and mailing lists.
• Handles MIME encoded messages and attachment.
• Handles UUEncoded messages.
• Two Admin Control Panels for all settings.
• Automated installation.
• Correct thread implementation. All imported messages and posts sent to newsgroups will be threaded correctly according to References and Message-ID headers.
• Optional setting to thread by subject for mailing lists that strip references header.
• Supports TABLE_PREFIX.
• Imported messages will be searchable just like normal forum posts.
• Similar threads are built when the messages are imported if the system setting is on.
• Multiple attachments importing.
• Save attachment to database or file according to vB setting.
• Thumbnail created while importing image files.
• Multiple attachment post to newsgroup is also supported.
• Selectable X-No-Archive header honouring system.
• Light on system resources.
• Global Killfile to filter out unwanted messages. Killfiles can have OR and AND effectiveness.
• Strip footer by setting for incoming messages. Strips out repeated message footers.
• Built options to disable signatures and footers for out going messages.

Future development

• AdminCP interfaces. (completed)
• Installation/upgrade script. (completed, coded into AdminCP)
• Traditional "> " lead posting interface.

What do you need and do not need?

You do not need to edit vB php files if you use two bookmarks
You need to edit one vB php file to insert two lines of code if you want direct click links within AdminCP.
No need to run SQL.

Installation

Basically the AdminCP interfaces take care of installation as well as upgrade in the background. Back up your database first. The scripts can modify your database without alerting you!

Please follow the instructions in the readme.txt file for more details.

Support

Please post your support questions to this thread. Please check the FAQ in the third post within this thread first.

================== UPDATES ======================
Jul 31 2004 (release version 2.2.0)

• Provided option of not importing attachments
• Provided option of pause between messages
• Provided option of not sending messages to USENET
• Provided mechanism to skip importing a message if it was crashed on it
• Provided "Default Footer" option
• Provided strip_footer function for mailing lists
  (written originally by KevinM)
• Provided Killfile functions with Header/Body/AND/OR settings
• Provided ability to thread messages by subject for mailing lists
• Provided ability to use a single POP3 account for different mailing lists
  (use prefix)
• Merged KevinM's attachment routine, save thumbnail to hard drive
  as vB3.0.3
• Official support to have the script run by Scheduled Tasks
• Many bug fixes

Please note, when you manually run the script, you must turn on the debug mode. Otherwise nothing will be displayed on the screen.

To run as debug mode: http://forums.yourdomain.com/gateway.php?debug=1

25 Oct 2004

• Provided the option of sending posts using the real email address
• Provided the option of sending posts using a fixed email address for
  each group.
• Posts are now held back if they are in the moderating queue
• Scheduled task log is now working
• Provided the option of sending the debug messages to the vB3 cron log
• Bug fixes

31 Oct 2004

Fixed bugs running the gateway by the vB3 Scheduled Tasks

1 March 2004

Uploaded the new zip file. It contains one or two bug fixes.

[pco]

Because your PHP is compiled without imap which is needed for imap_mime_header_decode to work in gateway.php. Furthermore this hack doesn't handle utf-7/8, for that to work gatway.php needs to be modified further (for instance with one of the comments mentioned under imap_mime_header_decode in the PHP manual).

[SEO Report]

David_R,

(Sorry for the late reply)

Originally Posted by David_R

does this idea works for you ?

I didn't try it (not yet ). I'm not sure, I'm afraid it will crash my server.

Originally Posted by David_R

please post your comments if you are able to speed up this SLOW process anyhow.

I installed a private test forum and setup this hack on it (total of 3 different forums with their respective gateway.php instance running every hour), this way I could simulate multi-threading & hopefully speed up the import process. The msg imported into the test forum would be ported to the live one using impex (very fast, since both forums are located within the same machine)

I don't have enough data to make any conclusion, but as far as I can see, the speed is almost the same.

Perhaps the script is already running at maximum speed, or perhaps as the author suggested, the import is slow because of the similair threads thing, I don't know (I have a dedicated box with 100Mbps, my nntp provider doesn't limit download speed).

In my case, getting a dedicated box and switching to a better nntp provider helped me speed up the process.

HTH,
-k

[David_R]

Originally Posted by SEO Report

David_R,

(Sorry for the late reply)



I didn't try it (not yet ). I'm not sure, I'm afraid it will crash my server.



I installed a private test forum and setup this hack on it (total of 3 different forums with their respective gateway.php instance running every hour), this way I could simulate multi-threading & hopefully speed up the import process. The msg imported into the test forum would be ported to the live one using impex (very fast, since both forums are located within the same machine)

I don't have enough data to make any conclusion, but as far as I can see, the speed is almost the same.

Perhaps the script is already running at maximum speed, or perhaps as the author suggested, the import is slow because of the similair threads thing, I don't know (I have a dedicated box with 100Mbps, my nntp provider doesn't limit download speed).

In my case, getting a dedicated box and switching to a better nntp provider helped me speed up the process.

HTH,
-k

I have tried with 3 nntp providers on a Dual Xeon with 8 GB RAM dedicated box.
But no improvements. It takes more then 12 hours to import about 2,00,000 posts

[SEO Report]

Originally Posted by David_R

I have tried with 3 nntp providers on a Dual Xeon with 8 GB RAM dedicated box.
But no improvements. It takes more then 12 hours to import about 2,00,000 posts

Did you mean 2,000 or 2,000,000 (millions)?

2 Millions in 12h is great IMO. I have a dedicated box (3Ghz, 1GB RAM) and I'm averaging 200,000 *REAL* msg/day (I'm using GigaNews).

But, if you meant 2,000 (or 20,000)...then yes, it *is* slow.

Can you confirm?

Perhaps you should check with your nntp provider if they're limiting the speed at wich you can download msg.

-k

[irubin]

Hello

I'm using this hack with some high-volume groups. Most of the time it works ok, however there are some problems.

1) not all newsgroup posts are downloaded to forums.
i debugged this thing and i know why but don't know how to fix it

the post numbers on nntp server are not in correct order, replies have earlier numbers than the original post which starts a thread. This causes the replies to be inserted first as new thread and then the thread starting post to be inserted as another new thread. I can't think of anything else then running a script from time to time which would solve such things using msgids and references. Any other idea?

for example one of the threads have the following server post numbers:

616189
  616185
    616187
       616233
          616248

and the result is two threads,
one

616185
  616187

and second

616189
   616233
      616248

2) some of replies (on forums) to downloaded posts are not posted to newsgroup. Even though they have "isusenetpost=1" they do not appear on newsgroup. When i make "isusenetpost=0" in DB and change $nntp_settings['last_postid'] to include this post and run gateway.php the post is uploaded correctly. The problem is only for some of the posts not all and i did not find out yet anything that would give me a clue about this problem. Msgids and references are ok

Thanks

[lierduh]

How often does your script run? If a reply arrives at your NNTP server before the original message, then threading could be wrong. The chance for this to happen will be reduced greatly if the time between the running of the script is increased. Also you can try the option to thread your messages by subject instead of Msgids. There is an option in the settings. I can't remember how I implemented this, but it is worth a try.

I can't explain the second problem. You can turn on the logging by editing the setting at the beginning of gateway.php. It will email you the debug message each time the script is run. If for any reason the server rejects the message, the debug message should show the error message.

Originally Posted by irubin

Hello

I'm using this hack with some high-volume groups. Most of the time it works ok, however there are some problems.

1) not all newsgroup posts are downloaded to forums.
i debugged this thing and i know why but don't know how to fix it

the post numbers on nntp server are not in correct order, replies have earlier numbers than the original post which starts a thread. This causes the replies to be inserted first as new thread and then the thread starting post to be inserted as another new thread. I can't think of anything else then running a script from time to time which would solve such things using msgids and references. Any other idea?

for example one of the threads have the following server post numbers:

616189
  616185
    616187
       616233
          616248

and the result is two threads,
one

616185
  616187

and second

616189
   616233
      616248

2) some of replies (on forums) to downloaded posts are not posted to newsgroup. Even though they have "isusenetpost=1" they do not appear on newsgroup. When i make "isusenetpost=0" in DB and change $nntp_settings['last_postid'] to include this post and run gateway.php the post is uploaded correctly. The problem is only for some of the posts not all and i did not find out yet anything that would give me a clue about this problem. Msgids and references are ok

Thanks

[irubin]

Originally Posted by lierduh

How often does your script run? If a reply arrives at your NNTP server before the original message, then threading could be wrong. The chance for this to happen will be reduced greatly if the time between the running of the script is increased. Also you can try the option to thread your messages by subject instead of Msgids. There is an option in the settings. I can't remember how I implemented this, but it is worth a try.

I can't explain the second problem. You can turn on the logging by editing the setting at the beginning of gateway.php. It will email you the debug message each time the script is run. If for any reason the server rejects the message, the debug message should show the error message.

Thank you for your reply. Ny script runs once every hour. I don't think that changing to thread by subjects would help. Also searching the DB for subjects is too slow when there are nearly 400 000 posts. The problem is when you get messages from the server from $lastmsg to $max, replies get downloaded first and inserted as new thread, then the thread starting post is downloaded and inserted as another new thread. I don't think that threading by messages would delete the first thread and link it to the second thread.

Thanks

[Ian Cunningham]

What exactly are the isusenetpost and msgid fields needed for? They are indexes in the post thread, but I think it might be slowing down inserting new posts for the rest of the forum?

[sdsvtdriver]

Has anyone figured out or modded the hack to exclude certain forums from sending posts to the Usenet? I have some forums I'd like the users to be able to post to the Usenet on, but other's I wouldn't.

[DigitalCrowd]

This is an awesome script, BTW. My issue is that I have one forum with around 200,000 posts and it is SOOO SLOW for it to sync up that database. It even stops the entire forum from responding while it's doing it's thing. Once it gets to importanting, it is fine, but the initial scan or whatever it's doing, is dog slow.

Any ideas or solutions?

Also, when it comes to filtering, it doesn't seem to filter SUBJECTS. I'd really like for it to do that, doing a H or B for the filter doesn't seem to work for subjects.