Replace that line with the following:

$DB_site->query("UPDATE forum SET lastactivethread = LEFT('".addslashes(htmlspecialchars($threadinfo[title]))."',$lastactivethread_length) WHERE forumid = ".$threadinfo[forumid]);

It's the extra dot in front of the addslashes that's causing the problem.

Sorry about that. I've updated the docs.

It won't affect anyone who did an upgrade. The error wasn't there in the previous versions. Only those who have done a fresh install of 2.4 will see this error.

Thanks for this Gilby. I can actually make use of this myself. I wanted to include the php.net mailing list archive into my forums. It doesn't comply with the usual usenet style mail archives so this is ideal!

In my outgoing usenet messages I use the email address at forums@britishexpats.com. This means I get a whole bunch of replies to forum users that have posted questions sent to my default email. The header actually looks like this:

fastforward <forums@britishexpats.com>

So the user is easily identifiable. Does anyone think it's worth adding something that will collect all these messages, run them through the spam filter and forward them on to the member? Or is it likely to cause members to become irate if a bit of spam leaks through?

how about forwarding them to the user's pm box with the following bit added at the top:

"This was sent to you as a reply to a newsgroup post.. we can not accepted responsibility etc..."

Chris... I thought you were dead! Where have you been? On holiday?

I never thought of using the PM box. thats not a bad idea.

I have the email address set up to be an autoresponder to let the person know that that message was not received. Since the newsgroup that I am doing it on is also an email list, a lot of people just hit the reply all button to respond, so those really don't need to be passed on to the person it's being responded to. The autoresponder also puts in a little plug for my site :D and tells them to go to the site to send a PM.

Anyways, I think the best option (probably more complicated to do too :() would be to have an option in the profile on whether they want to have the email sent to them to be forwarded to an address they specify, put in their PMs, or neither.

I'm glad it's useful for you! If you want to add it to the distribution of the usenet hack (and possibly modify it further), feel free to do so.

I'm kind of curious....what kind of a load does this impose on a server?

It really depends on how many groups you are pulling and how busy they are. On one site I have about 12 groups. When that script runs you don't even notice it. It may as well not be there. On my other site I am pulling in 70 groups. With this one the load gets to 1.0 - 1.5 as it gets near the end of the run. Most of the work is spent building that damn searchindex table! I wish it just go away! :) Without indexing the posts the load would not be worth talking about.

I created this Manual Insertion hack that integrates into the vB admin control panel. This hack will let you select messages that are in the que to be inserted but don't have references matching any of the existing messages and are therefore stuck in the usenet_article table. Once selecting the messages, you can do the following:

• Delete
• Start a new thread
• Merge selected messages together into one thread
• Insert post(s) into existing thread

All actions, except delete, take effect once the newnews.pl script is executed.

The installation is pretty simple, there is one SQL query that needs to be run and then two additions to to the usenet.php file. If you have any questions or find a bug, please feel free to ask.

fastforward, feel free to include this in the Usenet Gateway hack distribution or do anything else you want to with it.

I updated the mail2forum.pl script, the current one can be downloaded here or above.

Improvements

• Better at finding references in messages from mail programs that don't use the normal "References:" header and checks the "In-reply-to:" header which was commonly used many years ago.
• Improved getting all the references in the usenet_ref table, which it didn't do before.
• Checks date and doesn't allow a date to be in the future.
• Looks for "Posted-Date:" header as a higher priority than "Date:" as some posts didn't have a year in the "Date:" header (and did in the Posted-Date: header).

This fix entails two more code changes, both in 'postings.php'.

It corrects a duplicate bug when trying to move a thread with a redirect, or copying a thread.

No changes are required in any of the other usenet files or tables.

The changes are the last two (20 & 21) in vB_code_changes.txt.

Here's the changes to save downloading again:

20) postings.php - line 333

OLD CODE
--------
$DB_site->query("INSERT INTO thread (threadid,title,lastpost,forumid,pollid,open,reply count,postusername,postuserid, lastposter,dateline,views,iconid,visible) VALUES (NULL,'".addslashes($threadinfo[title])."','".addslashes($threadinfo[lastpost])."','".addslashes($threadinfo[forumid])."','".addslashes($threadinfo[threadid])."',10,'".addslashes($threadinfo[replycount])."','".addslashes($threadinfo[postusername])."','".addslashes($threadinfo[postuserid])."','".addslashes($threadinfo[lastposter])."','".addslashes($threadinfo[dateline])."','".addslashes($threadinfo[views])."','".addslashes($threadinfo[iconid])."','".addslashes($threadinfo[visible])."')");

NEW CODE
--------
// START USENET HACK
$DB_site->query("INSERT INTO thread (threadid,title,lastpost,forumid,pollid,open,reply count,postusername,postuserid, lastposter,dateline,views,iconid,visible,isusenetp ost,regpost,msgid) VALUES (NULL,'".addslashes($threadinfo[title])."','".addslashes($threadinfo[lastpost])."','".addslashes($threadinfo[forumid])."','".addslashes($threadinfo[threadid])."',10,'".addslashes($threadinfo[replycount])."','".addslashes($threadinfo[postusername])."','".addslashes($threadinfo[postuserid])."','".addslashes($threadinfo[lastposter])."','".addslashes($threadinfo[dateline])."','".addslashes($threadinfo[views])."','".addslashes($threadinfo[iconid])."','".addslashes($threadinfo[visible])."','$threadinfo[isusenetpost]','$threadinfo[regpost]','$threadinfo[threadid]')");
// END USENET HACK

21) postings.php - line 341

OLD CODE
--------
$DB_site->query("INSERT INTO thread (threadid,title,lastpost,forumid,pollid,open,reply count,postusername,postuserid, lastposter,dateline,views,iconid,notes,visible) VALUES (NULL,'".addslashes($threadinfo[title])."','".addslashes($threadinfo[lastpost])."','".addslashes($threadinfo[forumid])."','".addslashes($threadinfo[pollid])."','".addslashes($threadinfo[open])."','".addslashes($threadinfo[replycount])."','".addslashes($threadinfo[postusername])."','".addslashes($threadinfo[postuserid])."','".addslashes($threadinfo[lastposter])."','".addslashes($threadinfo[dateline])."','".addslashes($threadinfo[views])."','".addslashes($threadinfo[iconid])."','".addslashes($threadinfo[notes])."','".addslashes($threadinfo[visible])."')");
$newthreadid=$DB_site->insert_id();

$DB_site->query("UPDATE thread SET notes='".addslashes($threadinfo[notes])."',forumid='".addslashes($forumid)."' WHERE threadid='$threadid'");

$posts=$DB_site->query("SELECT * FROM post WHERE threadid='$threadid'");
while ($post=$DB_site->fetch_array($posts)) {
$DB_site->query("INSERT INTO post (postid,threadid,username,userid,title,dateline,at tachmentid,pagetext,allowsmili e,showsignature,ipaddress,iconid,visible,edituseri d,editdate) VALUES (NULL,'$newthreadid','".addslashes($post[username])."','".addslashes($post[userid])."','".addslashes($post[title])."','".addslashes($post[dateline])."','".addslashes($post[attachmentid])."','".addslashes($post[pagetext])."','".addslashes($post[allowsmilie])."','".addslashes($post[showsignature])."','".addslashes($post[ipaddress])."','".addslashes($post[iconid])."','".addslashes($post[visible])."','".addslashes($post[edituserid])."','".addslashes($post[editdate])."')");
}


NEW CODE
--------

// START USENET HACK
$DB_site->query("INSERT INTO thread (threadid,title,lastpost,forumid,pollid,open,reply count,postusername,postuserid, lastposter,dateline,views,iconid,notes,visible,isu senetpost,regpost,msgid) VALUES (NULL,'".addslashes($threadinfo[title])."','".addslashes($threadinfo[lastpost])."','".addslashes($threadinfo[forumid])."','".addslashes($threadinfo[pollid])."','".addslashes($threadinfo[open])."','".addslashes($threadinfo[replycount])."','".addslashes($threadinfo[postusername])."','".addslashes($threadinfo[postuserid])."','".addslashes($threadinfo[lastposter])."','".addslashes($threadinfo[dateline])."','".addslashes($threadinfo[views])."','".addslashes($threadinfo[iconid])."','".addslashes($threadinfo[notes])."','".addslashes($threadinfo[visible])."','$threadinfo[isusenetpost]','$threadinfo[regpost]','".$threadinfo[threadid] . addslashes($threadinfo[msgid])."')");
$newthreadid=$DB_site->insert_id();
$DB_site->query("UPDATE thread SET notes='".addslashes($threadinfo[notes])."',forumid='".addslashes($forumid)."' WHERE threadid='$threadid'");
$posts=$DB_site->query("SELECT * FROM post WHERE threadid='$threadid'");
while ($post=$DB_site->fetch_array($posts)) {
$DB_site->query("INSERT INTO post (postid,threadid,username,userid,title,dateline,at tachmentid,pagetext,allowsmili e,showsignature,ipaddress,iconid,visible,edituseri d,editdate,msgid) VALUES (NULL,'$newthreadid','".addslashes($post[username])."','".addslashes($post[userid])."','".addslashes($post[title])."','".addslashes($post[dateline])."','".addslashes($post[attachmentid])."','".addslashes($post[pagetext])."','".addslashes($post[allowsmilie])."','".addslashes($post[showsignature])."','".addslashes($post[ipaddress])."','".addslashes($post[iconid])."','".addslashes($post[visible])."','".addslashes($post[edituserid])."','".addslashes($post[editdate])."','$post[postid]')");
}
// END USENET HACK

To allow upgrade to beta 4 before I get the new release out you must do the following:

Delete the unique indexes on the msgid column in the post table and the thread table.

The new usenet gateway release will not require these indexes. However, to avoid any chance of duplicate usenet messages in the meantime, make sure you don't alter the message numbers in the control panel.

After upgrading to beta 4, you should be able to continue collecting news, but you won't be able to post outgoing messages.

I'll get the new release out in a day or two.

This is a drop in replacement for newnews.pl. It will allow you to continue collecting news without fear of duplicate posts.

Before upgrading to vB beta 4 you should drop the unique indexes on the msgid column in the post and thread tables. Then create the new usenet_loader table. I have included an SQL script that does this for you.

This new 'newnews.pl' script will eliminates the need for them.

Fastforward, thanks for the beta4 preparations!

I almost forgot. You should also run these two sql statements to create non-unique indexes for the ones you just deleted. These aren't essential, but the columns are used in joins during a news load. You will eliminate full table scans and reduce server load during news pulls if you create them.

This release eliminates the need for unique indexes on the msgid columns in the post and thread tables. This will mean there should be no problems when upgrading to future versions of vB (Other than the inability to post news obviously).

All the code changes have been revisited and changed where necessary for beta 4.1. The beta 4.1 version this was tested with is the one dated 4/4/01 at 4:49pm. There have been at least two builds with different timestamps since 4.1 was released.

The link in the first post of this has been updated to point to the latest version

Let me know if you have any problems.

I think if you release a new version (e.g. 4.0 after 3.x) then you should open a new thread, becuase for me it was confusing to read all the problems from Version 1.x to now.

I read through this whole thing, but there is one point I missed - please explain to a total usegroup newbie: Do I have to pay for usegroup access? (somebody mentioned something like that)
If so, which services are out there?

Thanks

It depends on what your webhosting provider offers. If they are dial up users, then you will probably be able to use their news server at no charge. You may be able to get through from your ISP as well by using your username and password to access that news server. If you don't have one of those, then you'll have to find one of the services that you'd need to pay to use... though, I think there are a few free ones out there.

I found a bug happens when a post is made on the forums and sent to the newsgroup. Then, when the post is fetched, the postid and threadid that is sent in the headers of the message are read in, but gets associated with the post that has the message number one more than the forum post. This results in the post in the forum getting the wrong message-id asigned to it, and then the post right after the forum post can't be inserted because the message-id already exists.

If the "Re-import Local Posts" is set to yes, then the forum post gets duplicated in the forums, otherwise it just gets deleted from usenet_article (or never inserted to begin with).

This occured in version 2.4 for vB 2.0 beta 3 and I upgraded everything to the new versions (2.5 and vB 2.0 beta 4.1) without any other modifications to the scripts.

I tried to look at the newnews.pl script but I was unable to see where there was anything wrong (though I don't follow that programming style all that well :rolleyes: ). I also noticed that when the newnews.pl file was run, and there were 2 new posts on the server, it would get 3 posts. This would be the case any time there were new posts where it would get one more than what was new, and if there were no new posts, it would fetch no posts. I thought this might have some relevance.

I think I saw that when using the 'post' in the NNTP module, it returns the message-id, so that may be a different route to use for getting the message-id.

I saw that before. I thought I'd fixed it... but obviously not.
I don't see how that can be possible. It can't insert an article if the msgid already exists. It's obviously a direct result of the bug above. (Or is that your point?)
What are you trying to say? What's wrong with the programming style? eh?! ;)
This is probably just a problem with the 'for loop'. It derives it's count from the last message number in the forum, the batch limit in the control panel and the last number reported by the newsserver.
This is the way I expected it to work. That doesn't seem to be the case though. I couldn't get it to return anything. It may be a newsserver issue but even CPAN makes no mention of a return value. If you can confirm it returns the msgid, let me know. it would make things a lot easier and more reliable.

By the way; I've got another version of the script ready. This one has a few efficiency changes but the big change is that it now continues to load replies in the correct order until there are no more left. This avoids having to make multiple runs when collecting history and should also prevent the stray articles that may not get loaded. Here's the zip if you want to play with it. I've got it running on my server. I'll package it up properly after I've looked at these other bugs and beta 5 comes out.

The one big problem with the script at the moment is the orphan reply checking. It does a full table scan for each newsgroup every run. That means for my 70 newsgroups I'm doing 70 full tablescans! My poor little Freedom 300 can barely keep up. As the post table fills up, it gets worse (I've got 23,000 posts at the moment). I need to find another way to do it or maybe make it an option for individual newsgroups. Although you get a few orhans in every newsgroup, it's really only a big problem with the mailing archives. Any ideas?

Hi fastforward,

Can I replace the old ./newnews.pl with this new one and if so are there any changes I need to make besides just replacing the file?

Thanks,

Michael

Does spam filter work during posting? I'm a little affraid that opening posting features could attract spammers and get my news server blacklisted?

That's the point... one of the results of this bug.
I wouldn't say there is anything wrong.... just not a style that I can follow very well (with those variable variables and such). I am not all that good at Perl.

I saw it on a page from a google search. I don't know which module it was (I've seen both News::NNTPClient and Net::NNTP which apear to have the same functions). On CPAN, neither of these modules mention anything about returning the message-id. :(

I did a lot of manual insertion to minimize how many articles were in the usenet_article table, then when I gave up for the itme being on inserting those posts, I renamed the table and created a blank one. Though that's just a temporary fix. Another idea is when not importing old archives, to limit the subjects it can match to those threads that have posts from the last 30 days in them. Maybe even make another table with just the subjects and refs of the threads from the last 30 days.

No it doesn't. I was thinking of adding it though. It's only a minor change so I'll add it in the next release with beta 5.

You should be able to do that. provided you deleted the indexes and added the extra table from the latest full release? If not then it won't work. Let me know if you need help.

Fastforward, your 2.5 vb code change instructions don't have any of these postings.php changes. Are they still needed?

No... not needed because the unique constraint has been removed from the msgid columns in the post and thread tables.

Make sure you deleted the unique indexes and re-created normal indexes on those columns.

This release is for vB 2.0 beta 5

Upgrading:

• The only vB code changes that differ to the previous version are in showthread.php. The way the Registered: and Location: bits are handled (via custom fields) make it messy to try to hide via code. If you removed the words from your templates in a previous version, you'll have to put it back again as this release doesn't attempt to hide it. If you really need them hiding, the best bet is to create a seperate style set for usenet forums. I'll look at adding options to handle it via the control panel in a later release.
• No DDL changes have been made.
• There is one DML change that deletes an unnecessary option. (The 'reply iterations' option that is now handled automatically).

This is all listed in UPGRADE.txt

INSTALL.txt details a fresh installation on a virgin vB 2.0 beta 5.

Download

All links in this thread have been updated to point to the new version. The latest version will always be in the first post of this thread.

I hope to get another release of newnews.pl out over the weekend that will add some of the outstanding features listed at the top of this thread. This should simply be a drop in replacement.

does anybody know some of the free ones?

You're so fast forward :) I only knew about beta5 from your post!

wow it amazes me how far this hack has come - it is a program in its own right now! Brilliant.

I can't wait to install, going to wait until stable vbeta comes out - the upgrading is killing me, i am only on beta3 i think the thought of all the template changes gives me sleepless nights :)

Cheers so much for the hack - so cool!

I totally agree!

I'm dead impressed with this script - it has added a great deal to the forums on my site.

Keep up the good work...just waiting for VB 2 to come out of beta so I can stop upgrading this damn thing!!

I'm trying to learn more about this hack before implmenting it on my own Forums.

My understanding is that any post that is made in the newgroup that you have "registered" to will show up as a post in that particular Forum in your Forums? Thos posts are by "unregistered" users who posted through the newgroup?

Also, usenet newsgroups receive a ton of spam/junkmail from people advertising everything from get rich quick schemes to porn sites. Is there a way to keep these kinds of posts out of your forum/usenet gateway???

There is fantastic, fully customisable spam filter built into the hack to prevent getting spam from --> newsgroups.

I've run into a problem...

I've just upgraded my VB to 2.0b5 and at the same time upgraded to Usenet Gateway 2.6 for b5...

Now all seemed to go well until I started pulling new posts again...i've got an extract below, where the newnews.pl just hangs after checking for orphans...the weird thing is that top still reports that mysqld is running and pulling quite a load on the server, so something is happening!...HELP

What version of the usenet hack were you running before? Did your previous version have the orhan checking routine?

If it looks like MySQL is doing something, it most probably is. The orphan checking routine is quite intensive if there are a lot of articles in the usenet_article table.

Assuming you didn't have the orphan reply check in your previous version:
1) If there are a lot of records in the usenet_article table (over 1,000), you can try putting an index on the 'title' column in the thread table temporarily. You should also investigate why there are so many records if this is the case. Make sure your 'expire' setting is set to something sensible in the control panel. It should be 2 to 5 days.

If your previous version did have the orphan checking routine:
The change in this version is the use of LOW_PRIORITY on the insert statement of that routine. It will take longer than usual, but I must admit, on my shared virtual server I did not notice any difference.

Finally, you may want to experiment by removing the STRAIGHT_JOIN hint from line 407 of newnews.pl. This hint forces MySQL to join the tables in the same order as the select clause. It works faster on my setup with that hint, but you never know, try removing it and see what happens.

Other than that, I don't know what to suggest. You're not getting an error message and that particular routine is doing nothing fancy. Just a bunch of querie on the title field of the thread table.

Let me know how you get on.

Stephan

Did the script ever finish? Did you kill it? Is it working now? Did your server meltdown? :)

A simple search on google resulted in finding this list: http://freenews.maxbaud.net/

They have a lot listed there, the free servers are probably not going to be very reliable though. You should try to get one close to where your server is located as it will be faster. There are lots of other lists out there too.

Fastforward,

Sorry for the delay in replying, I (quite stupidly) did the upgrade quite late in the evening and when I encountered the problems just killed the whole process and re-started it again this morning...

Anyway...

I was running 2.4 for B3, but I downloaded the 2.5 and 2.6 versions of the usenet hack and applied each SQL upgrade instructions seperately - so as if I had upgraded to 2.5 then 2.6...

I checked the usenet_article table and found over 3000 articles in there...since this is a beta test for my users I decided to empty that table and all other tables apart from the tables that seem to have settings in them.

I re-ran the script on the one newsgroup (that I know only has a few hundred posts in anyway) and the whole thing went really quickly...i'm doing some more testing and starting to re-populate the forums now so will post more details on how I get on later...:D