Usenet gateway

THIS THREAD IS CLOSED!
Posts in here will NOT be answered

This hack has undergone many major changes since this thread was started. Consequently, most of the posts have become dated and of little use. To coincide with the latest major release (20010712), a new one has been started.

See this thread for the latest version and discussion

I updated the mail2forum.pl script, the current one can be downloaded here or above.

Improvements

• Better at finding references in messages from mail programs that don't use the normal "References:" header and checks the "In-reply-to:" header which was commonly used many years ago.
• Improved getting all the references in the usenet_ref table, which it didn't do before.
• Checks date and doesn't allow a date to be in the future.
• Looks for "Posted-Date:" header as a higher priority than "Date:" as some posts didn't have a year in the "Date:" header (and did in the Posted-Date: header).

This fix entails two more code changes, both in 'postings.php'.

It corrects a duplicate bug when trying to move a thread with a redirect, or copying a thread.

No changes are required in any of the other usenet files or tables.

The changes are the last two (20 & 21) in vB_code_changes.txt.

Here's the changes to save downloading again:

20) postings.php - line 333

OLD CODE
--------
$DB_site->query("INSERT INTO thread (threadid,title,lastpost,forumid,pollid,open,reply count,postusername,postuserid, lastposter,dateline,views,iconid,visible) VALUES (NULL,'".addslashes($threadinfo[title])."','".addslashes($threadinfo[lastpost])."','".addslashes($threadinfo[forumid])."','".addslashes($threadinfo[threadid])."',10,'".addslashes($threadinfo[replycount])."','".addslashes($threadinfo[postusername])."','".addslashes($threadinfo[postuserid])."','".addslashes($threadinfo[lastposter])."','".addslashes($threadinfo[dateline])."','".addslashes($threadinfo[views])."','".addslashes($threadinfo[iconid])."','".addslashes($threadinfo[visible])."')");

NEW CODE
--------
// START USENET HACK
$DB_site->query("INSERT INTO thread (threadid,title,lastpost,forumid,pollid,open,reply count,postusername,postuserid, lastposter,dateline,views,iconid,visible,isusenetp ost,regpost,msgid) VALUES (NULL,'".addslashes($threadinfo[title])."','".addslashes($threadinfo[lastpost])."','".addslashes($threadinfo[forumid])."','".addslashes($threadinfo[threadid])."',10,'".addslashes($threadinfo[replycount])."','".addslashes($threadinfo[postusername])."','".addslashes($threadinfo[postuserid])."','".addslashes($threadinfo[lastposter])."','".addslashes($threadinfo[dateline])."','".addslashes($threadinfo[views])."','".addslashes($threadinfo[iconid])."','".addslashes($threadinfo[visible])."','$threadinfo[isusenetpost]','$threadinfo[regpost]','$threadinfo[threadid]')");
// END USENET HACK

21) postings.php - line 341

OLD CODE
--------
$DB_site->query("INSERT INTO thread (threadid,title,lastpost,forumid,pollid,open,reply count,postusername,postuserid, lastposter,dateline,views,iconid,notes,visible) VALUES (NULL,'".addslashes($threadinfo[title])."','".addslashes($threadinfo[lastpost])."','".addslashes($threadinfo[forumid])."','".addslashes($threadinfo[pollid])."','".addslashes($threadinfo[open])."','".addslashes($threadinfo[replycount])."','".addslashes($threadinfo[postusername])."','".addslashes($threadinfo[postuserid])."','".addslashes($threadinfo[lastposter])."','".addslashes($threadinfo[dateline])."','".addslashes($threadinfo[views])."','".addslashes($threadinfo[iconid])."','".addslashes($threadinfo[notes])."','".addslashes($threadinfo[visible])."')");
$newthreadid=$DB_site->insert_id();

$DB_site->query("UPDATE thread SET notes='".addslashes($threadinfo[notes])."',forumid='".addslashes($forumid)."' WHERE threadid='$threadid'");

$posts=$DB_site->query("SELECT * FROM post WHERE threadid='$threadid'");
while ($post=$DB_site->fetch_array($posts)) {
$DB_site->query("INSERT INTO post (postid,threadid,username,userid,title,dateline,at tachmentid,pagetext,allowsmili e,showsignature,ipaddress,iconid,visible,edituseri d,editdate) VALUES (NULL,'$newthreadid','".addslashes($post[username])."','".addslashes($post[userid])."','".addslashes($post[title])."','".addslashes($post[dateline])."','".addslashes($post[attachmentid])."','".addslashes($post[pagetext])."','".addslashes($post[allowsmilie])."','".addslashes($post[showsignature])."','".addslashes($post[ipaddress])."','".addslashes($post[iconid])."','".addslashes($post[visible])."','".addslashes($post[edituserid])."','".addslashes($post[editdate])."')");
}


NEW CODE
--------

// START USENET HACK
$DB_site->query("INSERT INTO thread (threadid,title,lastpost,forumid,pollid,open,reply count,postusername,postuserid, lastposter,dateline,views,iconid,notes,visible,isu senetpost,regpost,msgid) VALUES (NULL,'".addslashes($threadinfo[title])."','".addslashes($threadinfo[lastpost])."','".addslashes($threadinfo[forumid])."','".addslashes($threadinfo[pollid])."','".addslashes($threadinfo[open])."','".addslashes($threadinfo[replycount])."','".addslashes($threadinfo[postusername])."','".addslashes($threadinfo[postuserid])."','".addslashes($threadinfo[lastposter])."','".addslashes($threadinfo[dateline])."','".addslashes($threadinfo[views])."','".addslashes($threadinfo[iconid])."','".addslashes($threadinfo[notes])."','".addslashes($threadinfo[visible])."','$threadinfo[isusenetpost]','$threadinfo[regpost]','".$threadinfo[threadid] . addslashes($threadinfo[msgid])."')");
$newthreadid=$DB_site->insert_id();
$DB_site->query("UPDATE thread SET notes='".addslashes($threadinfo[notes])."',forumid='".addslashes($forumid)."' WHERE threadid='$threadid'");
$posts=$DB_site->query("SELECT * FROM post WHERE threadid='$threadid'");
while ($post=$DB_site->fetch_array($posts)) {
$DB_site->query("INSERT INTO post (postid,threadid,username,userid,title,dateline,at tachmentid,pagetext,allowsmili e,showsignature,ipaddress,iconid,visible,edituseri d,editdate,msgid) VALUES (NULL,'$newthreadid','".addslashes($post[username])."','".addslashes($post[userid])."','".addslashes($post[title])."','".addslashes($post[dateline])."','".addslashes($post[attachmentid])."','".addslashes($post[pagetext])."','".addslashes($post[allowsmilie])."','".addslashes($post[showsignature])."','".addslashes($post[ipaddress])."','".addslashes($post[iconid])."','".addslashes($post[visible])."','".addslashes($post[edituserid])."','".addslashes($post[editdate])."','$post[postid]')");
}
// END USENET HACK

[fastforward]

To allow upgrade to beta 4 before I get the new release out you must do the following:

Delete the unique indexes on the msgid column in the post table and the thread table.

The new usenet gateway release will not require these indexes. However, to avoid any chance of duplicate usenet messages in the meantime, make sure you don't alter the message numbers in the control panel.

After upgrading to beta 4, you should be able to continue collecting news, but you won't be able to post outgoing messages.

I'll get the new release out in a day or two.

[fastforward]

This is a drop in replacement for newnews.pl. It will allow you to continue collecting news without fear of duplicate posts.

Before upgrading to vB beta 4 you should drop the unique indexes on the msgid column in the post and thread tables. Then create the new usenet_loader table. I have included an SQL script that does this for you.

This new 'newnews.pl' script will eliminates the need for them.

[tamarian]

Fastforward, thanks for the beta4 preparations!

[fastforward]

I almost forgot. You should also run these two sql statements to create non-unique indexes for the ones you just deleted. These aren't essential, but the columns are used in joins during a news load. You will eliminate full table scans and reduce server load during news pulls if you create them.

ALTER TABLE post ADD INDEX(msgid);
ALTER TABLE thread ADD INDEX(msgid);

[fastforward]

This release eliminates the need for unique indexes on the msgid columns in the post and thread tables. This will mean there should be no problems when upgrading to future versions of vB (Other than the inability to post news obviously).

All the code changes have been revisited and changed where necessary for beta 4.1. The beta 4.1 version this was tested with is the one dated 4/4/01 at 4:49pm. There have been at least two builds with different timestamps since 4.1 was released.

The link in the first post of this has been updated to point to the latest version

Let me know if you have any problems.

I think if you release a new version (e.g. 4.0 after 3.x) then you should open a new thread, becuase for me it was confusing to read all the problems from Version 1.x to now.

I read through this whole thing, but there is one point I missed - please explain to a total usegroup newbie: Do I have to pay for usegroup access? (somebody mentioned something like that)
If so, which services are out there?

Thanks

[Gilby]

Originally posted by robertusss
I read through this whole thing, but there is one point I missed - please explain to a total usegroup newbie: Do I have to pay for usegroup access? (somebody mentioned something like that)
If so, which services are out there?

It depends on what your webhosting provider offers. If they are dial up users, then you will probably be able to use their news server at no charge. You may be able to get through from your ISP as well by using your username and password to access that news server. If you don't have one of those, then you'll have to find one of the services that you'd need to pay to use... though, I think there are a few free ones out there.

[Gilby]

I found a bug happens when a post is made on the forums and sent to the newsgroup. Then, when the post is fetched, the postid and threadid that is sent in the headers of the message are read in, but gets associated with the post that has the message number one more than the forum post. This results in the post in the forum getting the wrong message-id asigned to it, and then the post right after the forum post can't be inserted because the message-id already exists.

If the "Re-import Local Posts" is set to yes, then the forum post gets duplicated in the forums, otherwise it just gets deleted from usenet_article (or never inserted to begin with).

This occured in version 2.4 for vB 2.0 beta 3 and I upgraded everything to the new versions (2.5 and vB 2.0 beta 4.1) without any other modifications to the scripts.

I tried to look at the newnews.pl script but I was unable to see where there was anything wrong (though I don't follow that programming style all that well ). I also noticed that when the newnews.pl file was run, and there were 2 new posts on the server, it would get 3 posts. This would be the case any time there were new posts where it would get one more than what was new, and if there were no new posts, it would fetch no posts. I thought this might have some relevance.

I think I saw that when using the 'post' in the NNTP module, it returns the message-id, so that may be a different route to use for getting the message-id.