NNTP Gateway (Usenet/Newsgroups)

This hack provides a gateway to any NNTP newsgroup such as those from Usenet and other newsgroup servers. Selected newsgroups are imported from your news server and post on the forum are sent out to the newsgroup. This hack was greatly inspired by fastforward's usenet gateway hack. This one is a complete rewrite of that hack written in PHP and works with vB 2.2.2.

Installation:
Installation is pretty simple. No modifications to the vB php files are needed and it simply consists of dropping in the files to your forums directory, running the SQL modifications, and modifying the newsgroup settings for your desired newsgroups. For the complete installation details, read the readme.txt file.

Here are some of the features:

• Real cool vbcode to text conversion!
• All posts are inserted, even if parent thread cannot be found for replies
• Guests posts also get sent to the newsgroup.
• Importing of attachments from the newsgroup
• Poll info sent to newsgroup
• Attachment link sent to newsgroup
• Supports unlimited newsgroups and servers.
• email notification of new replies, just like regular forum posts
• Can import HTML messages and convert to limited vb code.

For control panel integration, please check out GameCrash's Graphical Interface addition.

What's not in it that was in fastforward's usenet gateway hack:

• No control panel integration at the moment. (Available separately)
• Does not support separate footers
• Does not allow custom email address to be used via the users option
• No spam control or string replacement support.
• No purging or autoexpire.

There is still lots that I'd like to add to it, but this is competely functional for basic newsgroup gateway support. It is most likely not free of bugs. If you have questions, suggestions or annoyances, please post in this thread.

vb3 support
lierduh modified this to provide support for the beta versions of vb3. When the release candidates of vb3 come out, I plan to add support for vb3 and add other features to this hack. If you have suggestions for new features, let me know.

E-mail gateway support
New in this version, you can now gateway with an email list. This will fetch emails from a POP email account and send via php's mail function to the mailing list's email address. To set up the list, in the newsgroup field, enter in the email address to send outgoing posts to. In the server field, enter in the pop3 server, and in the username and password fields enter in, you guessed it, the username and password to your pop account. Note: the email gateway capabilities has not been fully tested, so it might not work with your POP server. Also, it does not work in combo with a news<->forum gateway in the same forum (you can use either kind though in differnet forums within your forums).

[wolfman]

Thanks for this hack

[irubin]

Hello,

I have been recieving the following error when gateway.php attempts to post articles:

Posting Message from user: 441 (607) Article Rejected -- The 'References:' field requires one or more values

Any ideas?

Roy

[wolfman]

I have this installed and working great, once I got all the posts downloaded.

I just need to know if there is a way to keep these posts from showing up when a user clicks on the View new post link.

They like having the access to the groups but want to be able to use it more as an additional resource to search.

[liquidblaze]

i installed the hack and apparently all is working fine, i just ran gateway.php for the first time (still runnning - since 3 mins now) and i was asking myself how high the server load is if i run this script and for the crontab, what would be a reasonable time frame? i guess 10minutes is way too much

[allan grossman]

Originally Posted by wolfman

I have this installed and working great, once I got all the posts downloaded.

I just need to know if there is a way to keep these posts from showing up when a user clicks on the View new post link.

They like having the access to the groups but want to be able to use it more as an additional resource to search.

There is, wolfman

I made a search button that excludes Usenet posts. Pretty easy, really - on my system I made a button that linked to

../forumdir/search.php?s=&action=getlatest&forumid=4,5,8,9,12, 14,19,22,23

Substitute my forum numbers for the ones you want to search.

cheers -

[liquidblaze]

Originally Posted by allan grossman

There is, wolfman

I made a search button that excludes Usenet posts. Pretty easy, really - on my system I made a button that linked to the search

hmm didnt even think about this but im gonna integrate it

[allan grossman]

Originally Posted by liquidblaze

i installed the hack and apparently all is working fine, i just ran gateway.php for the first time (still runnning - since 3 mins now) and i was asking myself how high the server load is if i run this script and for the crontab, what would be a reasonable time frame? i guess 10minutes is way too much

Once you get the newsgroups populated server load is negligible. I gateway three relatively low-traffic groups every 15 minutes (~100 posts a day) and the script takes less than ten seconds to execute.

[lierduh]

I initially did a bit of hacking to this srcript regarding threading problem. You can see the post here:

https://vborg.vbsupport.ru/showpost....&postcount=587

allan grossman did a great follow up about this at:

https://vborg.vbsupport.ru/showpost....&postcount=635

After moving to vB V3 plateform, I have found the gateway is not compatible with V3. Below is my note to get the script to work with V3. I attached the hacked gateway.php with this post, there are quite a few places I need to fix, so a "diff" may not be a good way to see.

At the moment, I can only confirm my site is working with the Threading References problem fixed. A forum member can now also disable the signature when post to the gateway by uncheck "Show Signature". I must also point out: this is a quick hack to get my site up and running, so it may not be optimused nor it can be considered even near stable. You use it at your own risk. Some of the things I can think of after using this include: completely destroy your data, cause nasty replies from normal USENET users, the server may blow up...

===============================
For installation, please follow the Readme file contained in the zip file.
===============================

Edit: Fixed extra long References limitation.
Edit: Dec 4, fixed a typo in SQL string regarding attachment, possibly inherited from the original gateway.php. Please note, I have not been bothered to make the attachment fully working. Uploading fixed gateway.php
Edit: Dec 19 2003, uploading a new gateway.php which works with v3 Gamma. Modified the way of how to separate the new local posts and the new usenet posts.
Edit: Jan 21 2004, confirm my Gamma version of gateway.php also works for RC2.
Edit: Jan 23 2004, removed the older attachment gateway.php which was only good for vB3 Beta 7.

Please note, my current patched gateway.php does not work with "tableprefix". ie. if you defined tableprefix within config.php.

Rewrote the install instruction to save a lot of confusion.

Edit: Feb 2 2004

*Made "Xxxx Wrote:" working when reply/quote to a post.
*Added the ability to strip [font=xxx] BB tags.

Edit: Feb 4 2004

*Fixed the wrong thread link problem for the "goto newpost" at the forum list
*Uploaded the fixed gateway_RC2.zip

Edit: Feb 12 2004

*Added ability to strip nested BB tags.
*Uploaded a newer gateway_RC2.zip file.

Edit: Mar 2 2004

*Added ability to handle multiple attachments as in vb3.
*Uploaded gateway_Mar2.zip

(I have not tested this, but you should use this version if you want the attachment link(s) to work.)

Now confirmed the attachment is working. Removing older gateway.php

Edit: Mar 24 2004

*Discovered search index no longer worked with vB3 Gold due to a changed function name
*Uploading a fixed gateway.php (filename gateway_Mar24.zip)

Edit: May 12 2004

* I have done some major changes to the scripts. Improvement include:

== TABLE_PREFIX is now supported.
== Messages imported from usenet are now inserted into threads based
on Reference in the header instead of the subject.
== Built mechanism to avoid two instances of the gateway.php running
at the same time. This can easily happen if someone clicks the link in
the Who Is On Line page. This should also solve the problem of double
up posts and wrong post counters.
== Massive speed improvement!
== Cleaned up the codes.
== Implemented parentid for imported posts
== lastmessage is updated now after each message retrieval. This way
no message will be imported twice, even if the script is crashed at
some stage.
== Fixed a bug which occurs in long header lines. Message may not be
retrieved due to this bug.

This time instead of a quick hack, I have gone through almost each line. So it should be a much better script.

For people who has been using my previous gateway.php. It is strongly recommended to upgrade to this latest release. Please add the following records to your table:

INSERT INTO `nntp_settings` VALUES (NULL, 'Is Gateway Running', 'is_running', '0', 'Check if another instance of gateway.php is running', NULL, 0, 0);
INSERT INTO `nntp_settings` VALUES (NULL, 'Last Time Run', 'last_run', '1084316100', 'Last time the gateway.php was run', NULL, 0, 0);

Note for vB3. The thread view URL works with only "t=" instead as well as "threadid=". So you might like to change the 'threadurl' in the nntp_settings table. Check the attachment URL setting too. The format should be:

'http://forums.yourdomain.com.au/attachment.php?attachmentid='

Please also replace the nntp.php and mime.php with ones within the new zip file. I have made changes to nntp.php and mime.php files to fix bugs and make improvement.

(For people who downloaded the file earlier. Please download again. I made a stupid mistake in the gateway.php script.) The new file name is vb3_NNTP_Gateway_May12a.zip note the "a" in the file name)

Edit May 14 2004

**Build X-No-Archive header honouring system
**Improved speed
**Bug fixes
**Verbose output when run manually. Easier to find problem when interating with NNTP server.

change the honor_no-archive record from 0 to 1 to stop importing messages with X-No-Archive header, just as Google Group does.

For upgrade from the May12 release. Please replace all php files and insert two records into nntp_settings table

INSERT INTO `nntp_settings` VALUES (NULL, 'Honour No-Archive', 'honor_no-archive', '0', 'Do not import the message if the message has the X-No-Archive header set to yes. Google Group honours this Non-standard header.', NULL, 0, 0);
INSERT INTO `nntp_settings` VALUES (NULL, 'Last Max postid', 'last_postid', '0', 'The max postid in post table when the script was last run', NULL, 0, 0);

Edit: May 15 2004

* Fixed typos in nntp.sql and gateway.php.
* Modified setlastmsg.php to work with current versions.
* Packaged setlastmsg.php into the zip file.
* Reworded readme file regarding lastmsg.

Edit: May 16 2004

I have released the gateway under vB3 section of the forums. Please use the new thread for support. Latest files will always be posted to the new thread.

https://vborg.vbsupport.ru/showthread.php?t=65152

[rishel]

Ok. Here is a problem when trying to use Giganews. Giganews boasts it has 50,000 supposed threads in a particular NG.

However, when we run the gateway to start importing ng threads, some are no longer available.

This causes a timeout on our server. So I downloaded a normal newsgroup reader: Agent and sure enough, some posts were missing from a year ago, and some were not.

Does anyone have a fix or a workaround for this. Our PHP.ini file has been modified for testing purposes to allow an unlimited amount of time, but this still does not work.

Any ideas?

[allan grossman]

I asked this in email but I'll ask it again here

Why do you want year-old newsgroup posts? With Usenet usually if you get the last month's posts that's more than anybody will ever read.

Plus - although you can prevent folks from searching the Usenet forums if you want they still get added to the database when you rebuild your search index. That's a heck of a lot of space for something nobody's gonna read