NNTP Gateway for Usenet ( Newsgroups ), Mailing Lists
 

The latest version: V2.3.2 (31 Oct 2004)

What is it?

This hack provides a gateway from vBulletin forums to the USENET. It allows forum members to read newsgroup messages as well as to send posts to the newsgroups. In a nutshell, it turns vBulletin forums into a basic function news reader.

For an added bonus, this package also supports mailing list archive and forum to mailing list gateway. The software retrieves messages from a pop3 account and import them into the forums threaded. It also allows a forums user to send messages to the mailing list by the forum posting interface.

Why?

The gateway will be appreciated by forum members who do not know how to set up a news reader, or people who do not have port 119 open in their network. A forum interface to read and post news are often considered to be better than a news reader.

The gateway adds more contents to your forums and allows the members to interact with wider audience. It is common to see new users signing up to use this feature.

History

This hack is basically a hacked vB2 NNTP gateway written by Gilby to work with vB3. The original hack can be found here:

https://vborg.vbsupport.ru/showthrea...7&page=1&pp=15


Features

Some of the features and improvement include:

• Handles both newsgroup posts and mailing lists.
• Handles MIME encoded messages and attachment.
• Handles UUEncoded messages.
• Two Admin Control Panels for all settings.
• Automated installation.
• Correct thread implementation. All imported messages and posts sent to newsgroups will be threaded correctly according to References and Message-ID headers.
• Optional setting to thread by subject for mailing lists that strip references header.
• Supports TABLE_PREFIX.
• Imported messages will be searchable just like normal forum posts.
• Similar threads are built when the messages are imported if the system setting is on.
• Multiple attachments importing.
• Save attachment to database or file according to vB setting.
• Thumbnail created while importing image files.
• Multiple attachment post to newsgroup is also supported.
• Selectable X-No-Archive header honouring system.
• Light on system resources.
• Global Killfile to filter out unwanted messages. Killfiles can have OR and AND effectiveness.
• Strip footer by setting for incoming messages. Strips out repeated message footers.
• Built options to disable signatures and footers for out going messages.

Future development

• AdminCP interfaces. (completed)
• Installation/upgrade script. (completed, coded into AdminCP)
• Traditional "> " lead posting interface.

What do you need and do not need?

You do not need to edit vB php files if you use two bookmarks
You need to edit one vB php file to insert two lines of code if you want direct click links within AdminCP.
No need to run SQL.

Installation

Basically the AdminCP interfaces take care of installation as well as upgrade in the background. Back up your database first. The scripts can modify your database without alerting you!

Please follow the instructions in the readme.txt file for more details.

Support

Please post your support questions to this thread. Please check the FAQ in the third post within this thread first.

================== UPDATES ======================
Jul 31 2004 (release version 2.2.0)

• Provided option of not importing attachments
• Provided option of pause between messages
• Provided option of not sending messages to USENET
• Provided mechanism to skip importing a message if it was crashed on it
• Provided "Default Footer" option
• Provided strip_footer function for mailing lists
  (written originally by KevinM)
• Provided Killfile functions with Header/Body/AND/OR settings
• Provided ability to thread messages by subject for mailing lists
• Provided ability to use a single POP3 account for different mailing lists
  (use prefix)
• Merged KevinM's attachment routine, save thumbnail to hard drive
  as vB3.0.3
• Official support to have the script run by Scheduled Tasks
• Many bug fixes

Please note, when you manually run the script, you must turn on the debug mode. Otherwise nothing will be displayed on the screen.

To run as debug mode: http://forums.yourdomain.com/gateway.php?debug=1

25 Oct 2004

• Provided the option of sending posts using the real email address
• Provided the option of sending posts using a fixed email address for
  each group.
• Posts are now held back if they are in the moderating queue
• Scheduled task log is now working
• Provided the option of sending the debug messages to the vB3 cron log
• Bug fixes

31 Oct 2004

Fixed bugs running the gateway by the vB3 Scheduled Tasks

1 March 2004

Uploaded the new zip file. It contains one or two bug fixes.

Very nice work - I'm yet to install, but I'm sure many members will find it useful as with the first version.

I reserve this post position for support FAQ. This way it will be easily found and referred to.

If you have problem retrieving and posting messages, please make sure your web server can access the NNTP server.

If you have shell access to your web server either by telnet or ssh. You can test the server access using:

=============================
First, ssh or telnet to your web server.

Then telnet the news server from your web server. For example:

[ns@mb eyoforums]$ telnet msnews.microsoft.com 119
Trying 207.46.248.16...
Connected to msnews.microsoft.com (207.46.248.16).
Escape character is '^]'.
200 NNTP Service 6.0.3790.1146 Version: 6.0.3790.1146 Posting Allowed
GROUP microsoft.public.access
211 13988 1 86055 microsoft.public.access
ARTICLE 86054
==============================
ie. after telnet to the NNTP server by using "telnet msnews.microsoft.com 119"
type
GROUP group.name
then type
ARTICLE xxxxx

(xxxx is the post number)

You should get the whole article including the header. If the telnet does not even work. There is no point trying to fix the gateway.

You can do other things while you are under the telnet session. Type "help" will get a list of the commands you can use. To quit a telnet session, type push Ctrl key together with "]" without the quotes.

Server Crashed during the creation of gateway tables, now I am getting MySQL errors when open AdminCP interfaces

The best way to do when this happens is to delete both nntp_groups and nntp_settings tables. Open the AdminCP interfaces. These tables will be recreated by the AdminCP interfaces. Tables will also be upgraded to the latest version automatically.

It tries to download instead of displaying the contents on the screen when I run it manually

Please fix the Windows registry following the instruction here:

http://support.microsoft.com/default...&Product=ie600

The gateway retrieves the messages our forums sent

This usually happens if the news server rewrite your Organization header, eg. to insert advertising etc.

Please set the "No Organization Check" to '0' without the quotes.

Which newsgroup providers do other forum owners use?

Please check out this thread:

https://vborg.vbsupport.ru/showthread.php?t=75232

To upgrade from the beta vB3 version in the vB2 hack thread to this new vB3 version, you need to upload the new files here, and then run these queries before running the new gateway to add the new nntp settings variables:

Otherwise, this new gateway.php won't work with the old settings. :)

Cool. Anything's better than google's interface.

What exactly do I need to be abl to run this? I use SSH to connec tto my Web server and while in the SSH session I tried the command telnet news.swiftel.com.au 119, but got the error bash: telnet: command not found.

That means your host has removed telnet program from your server, or disallowed the users to use telnet.

Above steps are only for fault finding. No need to try if you have no problem.

By the way, in the example, I used my own news server. You need to use your own.

lierduh's upgraded USENET hack seems to work a whole lot better than the old ones. :) No more blank threads, it actually works faster, and there is less load on the server. Thanks!

Hey, lierduh -

Suggestion for a future release:

I've commented out the entire signature routine in gateway.php to keep my users from being flamed hard by real Usenet users - since you have to be pretty durned old to remember when a signature was

<dash><dash><space><cr>
No more than
four
lines of
text

Setting a database field to skip the whole signature routine to keep Usenet neophytes from getting their feelings hurt might be kinda cool - but commenting out the routine doesn't take a whole lot of technical expertise.

Just a thought ;)

That should be quite easy to implement. At the moment, the users can uncheck the "Show your signature" when post. I can have a "disable signature" option in the settings table to disable the signature altogether.:)

Another thing I have on mind is to create a posting interface which behaves just like a news reader. ie. No BB codes available, just the "> > " for quoted text. It will be kind like WYSIWG of a news reader. This way, it will be much easier to trim the quoted text, and it will reduce the chance of forum news posts getting flamed. :devious:

Right now, I just want to get the second part of the Admin CP interface finished.

I have something to add to this discussion... something I never thought about until someone pointed it out. I have had this hack installed for at least a week and have never been able to get my posts out to Usenet. I just checked with my host and, sure enough, they have outgoing port 119 blocked! :ermm: To make matters worse, they refuse to open it up for my domain.

So, if anyone here is having problems with their posts not showing up, first thing I would do is check to see if your host is blocking outgoing port 119.

I'm screwed unless someone wants to code POP3 to NNTP or something to that effect. :ermm:

this sucks...

To those people who have this thing working, what hosts are you using? Mine is Lunarpages.

That won't help your situation, your web server needs to talk to a NNTP server on port 119 to send the news to it. pop3 to nntp is used to post news by email instead of vB.

If you can retrieve news but not post, that means the port is open. Does the news server accepts posts? or does the server accepts posts from your web server's IP range?

If you can ssh to your web server, just try to use one of the Unix news readers from the shell. eg. tin. You can even use lynx to read news. Check Google, I can't remember.:)

Open for incoming only! The server is firewalled and port 119 outgoing is blocked! From what the support staff has told me, there is no way in hell I am getting anything out 119... ever. This is their reply.

Some news providers will also let you run NNTP on other ports - like port 80. I'd contact my news provider.

At work the NNTP *protocol* is blocked :(

Good idea, Allan... I am going to look into it.

thanks!

Lierduh,
I'm glad to see you moving this project forward! I'm among the ones who have had your earlier version running for several weeks on my VB-3 forum and I am really looking forward to these updates, especially the admin section.

Thanks!

Lierduh,

Just got this up and running on a fresh install. Fantastic job. I've used the vb2 version for a year and this is a vast improvement. Particularly appreciated are the no-archive option and UU-encoded attachments (This now opens up all the binary groups which was too problematic with just mime encoded features). Control panel works great as well.

Two questions related to the attachments:

Is there a way of creating the thumbnails at the same time the the messages are downloaded? If you have the thumbnails option active, it shows the broken red x in the message. When you manually run the re-create thumbs option from the update counters menu it does create them, but it means re-creating the entire attachment table.

I can't get the attachments to work when using the 'attachments as files' option (this would be my preferred option with a vast number of attachments). the files download, but when you click the link on the post it can't find the file. When you reverse the process (i.e. switch back from files to db storage) the images reappear.

Thanks for a great upgrade.

Ok, so, for those who are following along with my connection "soap opera" woes, here's a bit of good news...

From Giganews...

I am dropping Newsguy just as soon as I get home. I asked them about port 80, but apparently I can only use 80 if I go with their $75.00/month blah blah plan. LOL! Um... NO THANKS!

For those who are having problems posting from VB to Usenet, take note, port 119 on your web server may be blocked for OUTGOING data.

I've been using this hack with Giganews for quite awhile. Works just fine :)

Take a look if you like - I've got one newsgroup set to autoprune every day so it only holds 90 days worth of posts, but it works just fine.

Forumhome is at http://ebassist.com/vb3

The only mod to the gateway I've made is to disable signatures in gateway.php - but we discussed that earlier in the thread :)

edit: Before you switch news providers it might be a good idea to see whether your ISP blocks only port 119 or the entire NNTP protocol. Telnetting to a news server won't tell you whether your ISP blocks NNTP, I'm afraid :(

Made the upgrades (including the added sql parts per Erwin, thanks) and the latest gateway hack is working just fine, I like the new interface in the admincp too.

Two questions;

1) Lierduh, when you talk about adding a 'new usenet posts' to the navbar, is that as part of the 'quick links' menu or is it a whole new button on the navbar? I need to add it to quicklinks because I have removed the other 'new posts' thing.

2) when Allan was talking about commenting out the entire signature routine in gateway.php, could someone show exactly what I need to comment out? I would definitely like to remove that part for now.

Thanks Lierduh!

FWIW, I am using EasyNews as a Usenet provider, seems to work pretty well.

have a couple of questions. I would like to disable all post from a couple of users that are in the usergroup. Is there away of deleting thread topics that start with certin users?

That is good. I have not actually tried a binary group (now I tell you.:))

I will have a look at this.

Sorry for my ignorance, where is this option located? are we talking about outgoing or incoming. I can check it out, but I am feeling lazy after working on the AdminCP interfaces.

"New Usenet Posts" will be at the main navigation bar next to the "New Posts" link if you followed the readme file. If you have moved the original link to quick link, you should be able to do the same for the USENET link.

The next version will contain the signature option.:)

Can't you disable them posting in the usenet forums altogether?

i am talking about newsgroup users that i don't want to display on my messageboard.

Update: With the AdminCP interfaces becoming available, this is no longer a problem. The first time the panel is openned, the script will detect and carry out the upgrade.:)

To disable the forum signature, find and remove this code in gateway.php:

The newsgroup signature (with links to your forum and the thread) will still be present. :) The large forum signatures will no longer be added.

Or - if you want to, just comment out the whole thing - change thisto thisor - you can remove the entire routine. I choose to leave the code in place - which causes a tiny performance hit, but at least it's easy to remember what I did when a new release comes out :)

cheers,

allan

i am getting this error below

I'd like to weigh in here if folks don't mind.

Except for the .moderated newsgroups, Usenet is completely unmoderated - you can certainly do whatever you like with your forum, but IM frequently less than HO moderating an unmoderated newsgroup goes pretty much against what Usenet is all about.

What I do is run a disclaimer as an announcement at the top of the Usenet forums that reads like this -

==========

Folks -

For those of you who are unfamiliar with Usenet, the best description I've seen is at

http://www.faqs.org/faqs/usenet/what-is/part1

and

http://www.faqs.org/faqs/usenet/what-is/part2

No one "owns" Usenet and the groups that are gatewayed into ebassist are completely unmoderated. The feed that we provide is a service to ebassist users, and is just a gateway into another discussion area. You're liable to see flaming, links to ads and porn sites and a whole bunch of really objectionable stuff if you look hard enough.

But, as you continue to look, you'll see some real gems of information and some *very* knowledgeable folks.

Once again, ebassist doesn't own Usenet, has no control over its content and the feeds are provided without any interference (or moderation) from the folks on ebassist dot com.

Usenet can be harsh at times - and it's definitely not for the fainthearted, but I think providing an uncensored newsfeed provides more of a service to the ebassist community than filtering content would.

To sum things up, internal forums on ebassist dot com are moderated. External feeds like Usenet are not. Read at your own risk :)

the management

==========

But - it's your board and your call ;)

In the 'attachments' menu on the control panel there is an 'attachments storage type' option, which enables you to store the attachments as files rather than in the db.

When this option is used, the link to the attachment in the post breaks (although the file still gets downloaded to the db, the script is searching for the file in folder ont eh server). Hope this makes sense.

No problem, have some sleep first.... :nervous: !

More information please. Is this the first time you run? how many posts roughly are we talking about importing? I have set the script maximum run time to 1800 seconds. If you can not import the whole lot, restart again, it will pick up where the last run left off. This is one of the features this new version has. Also how is your server load. Can it handle a lot of posts to be imported in one go? With each post imported, the server needs to build search index and it is resource intensive.

i have to run the script about 3 times and to get all of the information. I am still planning on adding more newsgroups to the site.


by the way i used the script to import about 30,487 post and it work fine. The only probem i had is everytime it did it is i have to change the "Is Gateway Running" option to 0 each time. My server load only went up to 2.0 which is 100% on both cpu.

lierduh, you've released 2 updates already. Did you make any changes to gateway.php, or are the updates purely for the AdminCP?

The two beta files are only the AdminCP interfaces. The main script has been not updated since the ___b.zip .

I will do something tonight to add the signature option routine and may be others.

A few questions....

I am still running vb2.3.0, and gilby's version of this hack, the reason for staying with vb2.3.0 is because of the lack of a 'suitable' nntp hack addon, which you have now provided, so as long as it works with what I am about to ask, I shall update my forums to vb3.0.1

WIll this allow/import/whatever gilby's database modifications? as is from gilby's own version for vb2

Also as gilby's version allows me to pull from both NNTP and a pop3 mail box (downloads posts etc from a mailing list), will this version also allow it?

I do not use the pop3 to post from the forums to the mailing list, but to pull from the mail server the posts and then put them into the forums.

If you want to know what I mean about pulling posts from a pop3 mail box, please go here, and view the subforums, I have and the comments below explaining what they are:

http://www.sanjiyan.co.uk/forums/for...php?forumid=35

Many thanks...

There is one extra field in the post table needed. The first time you open the AdminCP interface, it will detect what needs to be done and upgrade for you. This will also be the way it works with future upgrade.

I have disabled this because I have not checked whether it will work. Also no one else has mentioned the need to use this. I will have a look at it in the next few days and enable it in the next minor version. It is something pretty simple.

Thanks, because the pop3 function is something I rely very heavily on with this hack, as 40% of my need for this hack is for the pop3 mail pulling ability.

EDIT:

Also which file do I need to download? the nntp_gateway_v2.0.zip or vb3_NNTP_Gateway_May16b.zip

Not trying to be big headed here, but the reason this option was put into the Gilby hack, was mainly because I asked it to be, I think to date no one else uses that function in the hack bar me, but it is a very nice function to have.

Anyway, can not wait for you to get this option working again, THEN I will upgrade my boards.