Fastforward, my members didn't like the usenet messages being included into the "view new posts" search, as we have a much slower traffic, so I thought it might be useful option in the CP to include or exclude usenet messages from the search new posts function. Just an idea. I intend on hacking it for the mean time to exlude usenet messages.

Yes, I am using 2.3

I never really thought about that actually. But you're right that must be very annoying to see all those mesages :) I'll add it ASAP

Gilby, I'm still looking at your problem. In the meantime, have a look in the usenet_article table at the messages that are not being inserted and look at the refs column and the ord column. Then do a search in the post table and see if one of the refs matches an inserted post AND the ord is either equal to or 1 more than an inserted post.

Also, did the missing messages originate from usenet or your forum?

Well I've just tested it on that webdiscuss news server and I managed to get all the messages from that thread with no problems. They didn't all come in in the first run though. This is because it only does one 'insert into forum' batch each time. So although all messages were retrieved and placed into usenet_article during the first run, it took three more runs to insert them correctly in order. The only other explanation is if the references weren't placed in your outgoing messgages correctly. I'll check the code in the newreply.php and newthread.php again.

On another note. I have come across a problem with the control panel if gzip is not enabled or output buffering is not on. You'll get the 'Cant add header' errors. I now see why Jelsoft didn't refresh the control panel each time a change is made. I'll look into fixing that tomorrow.

Now I am puzzled. :confused: I did a clean install of the forums (imported the original DB that had no posts, etc. into a new DB) and imported the rec.sport.unicycling newsgroup and now I only have 15 messages in the usenet_article table and when I look at it, there are reasons that I see that they were not inserted. Stuff like "Re(2):" or "Re: [Re:" are in the subject and some of those have no refs (so the orphan replies code doesn't associate a ref for it). Earlier today, I had 52 messages in the usenet_article table and most looked like they shouldn't have had the problems. I also downloaded the latest zip of the hack and replaced the newnews.pl script with the one I had (which was version 2.3, but from before the addslashes() fix was put into the code replacements which I didn't think affected the newnews.pl file). Another thing I did was set it for just the one newsgroup without using the multiple servers since you mentioned the bug with that (although I never saw that particular bug, and you deleted it from your post). So, now it appears to work. Thanks for looking into this problem and sorry about trouble.

Anyways, I think a feature that would be nice to help with posts not being inserted would be to have a feature in the control panel where the posts that are still in the usenet_article table are listed and then have an option to choose a thread to put it into. This would help with the posts that have weird subject lines since one unrecognized post makes all the child posts get stuck as well.

More reports:

1. I got one duplicate message inserted (has same poster and title) but has no body:

Here's the empty one:

http://www.lowcarber.org/forum/showt...&threadid=1379

Here's the one that's not empty:

http://www.lowcarber.org/forum/showt...&threadid=1373

2. CP Stats includes usenet posts, which more than tripled my daily stats. This is just cosmetic suggestion to not include usenet posts in stats or give them their own stats.

3. For some reason the 2.3 newnews.pl doesn no pull news articles, it just displays connected and then "clean disconnect" When I run the 2.1 version it polls articles properly. I currently have the 2.3 installation of the hack, but my cron job uses the 2.1 version of newnews.pl.

This is probably a duplicate post on usenet aswell. There is nothing that can be done about that. The script should check to see if the message is completely blank and not insert it if it is.
Believe it or not, I've never actually looked at that option in the CP. I think I tried it once in beta 1 and it didn't work so I never looked again :) I'll fix that sometime soon.
This should not be happening. You may introduce other problems by using an old version of newnews.pl. Even if there are no messages it should tell you. The only reason I can think of is if all newsgroups are turned off in the control panel. Version 2.1 doesn't check for this but 2.3 does. If this is the problem give yourself a slap! :rolleyes:

It turned out the problem was actually caused by the server name not being saved. The default one is saved correctly, but any custom. per newsgroup entries were never inserted into the settings table. This was fixed a few minutes after I posted yesterday. It was just a small edit in the usenet.php file.

Slap! :p

You might want to keep an eye on this. If there really is a problem, what you are seeing indicates there is something wrong with usenet replies to forum originated messages. Although I've never exprienced any problems, there could be some kind of timing issue that doesn't apply the references correctly.

What happens when a forum message is destined for usenet is this:[list=1][*]A dummy msgid is created in the msgid column to avoid primary key violations[*]The message is sent to usenet at the next run. (this is the last thing it does after pulling articles. at this stage there is still only the dummy key available which is meaningless)[*] The next run retrieves the msgid of the posted message and updates the msgid column with the correct message id. (This is is the fisrt thing it does, so theoretically, it should have no problem with the messages it's about to pull)[/list=1]
So the bottom line is... it shouldn't be a problem, but there may be some scenario I've overlooked that could cause the msgid not to be updated.

The obvious cause is if you change your 'useragent' in the control panel. This is how the script identifies your messages. Once you have set it, you should not change it if you have outstanding messages from usenet.

Yesterday when I tried the newsgroup again from scratch, I did not have any better success than before, as I had thought. What happened was that I had the days to keep orphaned replies set to 7 days, so it deleted the ones that were older than 7 days. So today, I tried it with a setting that would be longer than any of the dates of the messages that are in the newsgroup and there were 69 that could not be inserted into the forums (and 468 that were put in the forums).

Anyways, looking at the ones that did not get inserted, I had these cases:

1. A thread starting post had two direct replies, both had the reds set to the msgid of the first post in the thread. However, the one that was not inserted had an order of 2 while the parent post had an ord or 0 and the other reply had an ord of 1. So that didn't fit the ref matching along with the ord being equal to or one more that the parent post. So it didn't get inserted. So, I changed the MySQL call in the newnews.pl file to ignore the ord. So on line 404, I removed "((d.ord = a.ord) OR (d.ord +1 = a.ord))"so that I now have:
   Code:

   ---

   my $q3 = db_fetch("SELECT b.title, a.nntpposter, a.forum, a.msgid, a.dtm, a.subject, a.poster, a.body, a.ord, a.postid, a.email, b.threadid, c.ref FROM usenet_article AS a, thread AS b, usenet_ref AS c, post AS d where b.threadid = d.threadid and b.forumid = $$newsgroup->{forumid} and c.ref = d.msgid and a.msgid = c.msgid  ORDER BY a.dtm, a.ord");

   ---

   That made it where this post was then inserted into the proper thread along with 4 others that had the same problem and brought the total in usenet_article to 64.
   
   The rest of the cases are now without the ord part of the MySQL query.
   
2. A message has two refs in it and both those refs are associated with posts in the post table, however, these referenced posts are in different threads. Also, the original message on the newsgroup does not have any references associated with it. The refs associated with this message both had the same title "(no subject)". So the orphaned replies code must have associated this post with both of these messages.
   
3. A reply to a message that had a different title than what it's reference was and then the post that was not inserted was a reply to that message that had no refs in the headers. The original post was from a newbie to newsgroups and replied to a message to start a new thread.
   
4. One post was a reply to a post, but the reference for that post did not match the parent post (the message did have quotes from the parent post though).
   
5. Some posts have weird subject lines associated with it, such as one person's reply to a message titled "My Post" ends up being "Re: [My Post]", and a reply to another replied message ends up being "Re: [Re: My Post]". These posts do not have an references in them on the newsgroup, . Also, there were instances of subjects like "Re(2): My Post" and "Re: Re(2):".
   
6. Some posts to the mailing list do not make it to the newsgroup, so the replies to that get trapped in the usenet_article table.
   
7. Some posts looked like they matched all the conditions to be inserted, with the refs being associated with the post(s) that id goes to. I went through the MySQL query and everything matched. So I added some console() calls to see if those messages did get selected and they did. So, I added some more console() statements to determine what's happening, so starting at line 440, I put in:
   Code:

   ---

                     console("\nTrying post from $poster:");
          if (db_execute("INSERT INTO post (allowsmilie,threadid,username,dateline,pagetext,visible,ord,msgid,userid,ipaddress,isusenetpost,seq) VALUES ($config{allowsmilies},$threadid,$poster,$dtm,$fbody,'1',$ord,$msgid,$userid,$nntpposter,1,$seq+1)",1)) {
            console(" posted!");
            $postid = $dbh->{'mysql_insertid'};
            db_execute("DELETE FROM usenet_article WHERE msgid = $msgid");
            db_execute("DELETE FROM usenet_ref WHERE msgid = $msgid");
            my $q4 = db_fetch("SELECT lastpost FROM thread WHERE threadid=$threadid");
            my ($lastpost) = $q4->fetchrow_array;
            if (!$lastpost) { $lastpost = $dtm; }
            db_execute("UPDATE thread SET replycount = $seq ".(($dtm >= $lastpost)?",lastpost=$dtm,lastposter=$poster":"")." WHERE threadid=$threadid");
            my $q5 = db_fetch("SELECT lastpost FROM forum WHERE forumid=$forumid");
            $lastpost = $q5->fetchrow_array;
            if (!$lastpost) { $lastpost = $dtm; }
            db_execute("UPDATE forum SET replycount=replycount + 1 ".(($dtm >= $lastpost)?",lastpost=$dtm,lastposter=$poster":"")." WHERE forumid=$forumid");
            indexpost($postid);
            push(@updated_threads,$threadid);
        }
                else { console($DBI::errstr); }

   ---

   This resulted in these results:
   Getting article batch from rec.sport.unicycling
   No new messages in rec.sport.unicycling
   inserting new threads into forums
   inserting replies into forums
   
   Trying post from 'Jonathan Marsha':Duplicate entry '<CSujlIAW5Mw6Eww0@jbmarshl.demon.co.uk>' for key 5
   Trying post from 'Mark Wiggins':Duplicate entry '<3AC0D05A.620D0A71@ftel.co.uk>' for key 5
   Trying post from 'Mark Wiggins':Duplicate entry '<3AC0D05A.620D0A71@ftel.co.uk>' for key 5
   Trying post from 'Mark Wiggins':Duplicate entry '<3AC0D05A.620D0A71@ftel.co.uk>' for key 5
   Trying post from 'Chuck Webb':Duplicate entry '<As6w6.166$9d.54655@newshog.newsread.com>' for key 5
   Trying post from 'Greg House':Duplicate entry '<5Xew6.151$pn3.483290@nntp3.onemain.com>' for key 5
   Trying post from 'Greg House':Duplicate entry '<5Xew6.151$pn3.483290@nntp3.onemain.com>' for key 5Processing outgoing messages
   Clean disconnection from news.tc.umn.edu
   
   Um, ok, now I figured that one out since those posts are already in the forum somehow ended up in the usenet_article table many times.

So in conclusion, the rec.sport.unicycling newsgroup is pretty messed up as far as the posts being properly threaded. The newsgroup is also an email mailing list so that causes some pretty strange things for referenences (such as no references) and the formatting of the subject line.

That isn't the problem that I have with the one newsgroup, rec.sport.unicycling, as I haven't ever posted a message that went to the newsgroup from the forums. It may have been a factor in the other newsgroups that I played around with but didn't focus too much on trying to get them to work.

I don't understand how the message with an order of 2 was a direct reply to the parent. If this were the case, it would have an order of 1. This 'ord' field is simply a count of references in the header. The other unlikely scenario is that the news server added two references (but as you pointed out, there was only one ref in each message). I will check the code, but as I'm sure you've seen, not a lot can go wrong with counting how many refs are in the header. Somewhere between the count and the insertion into the usenet_article table it must have lost one of the refs. :confused:

If you leave that clause out permanently, you will get messages out of synch. This will happen more on busy groups that are propogated to many servers.

I thought long and hard about the best way to handle it, and this method with the ord seemed to be the method that fit most situations. In addition to the ord column, there is the 'seq' column this is simply incremented and the thread id displayed on this order. The 'ord' method makes sure the 'seq' 'makes sense'.

As for the other issues, as you probably realize, there's not much we can do about that I don't think. (Except your suggestion of flagging un-inserted messages for manual insertion). If everybody used a good solid news client we'd be fine :)

I just noticed that on the usenet forum, the "Next thread" link doesn't show up properly. I'd keep pressing it for next thread, and I'd think I'm at the last thread since it only shows "Last thread" But if I check the thread list, there are a few more to read.

This happens on and off, they sometimes show up till the last thread?!?

This must simply be a post count problem. My script does nothing to the actual forum display or the way vB handles it. Try updating the threadcounts. I'll have a look at the code and see if there's a problem with incrementing and indexing of the posts in the script.

I've already noticed a small problem with the counting of usenet user posts if you have enabled the 'import user from usenet' option. That will be fixed in the next release.

I believe I've found the problem. It was updating the replycount in the forum table, but not the threadcount. In addition, it was updating the replycount twice; once for the thread starter and once again for the associated post. I'll post a new release later today along with a few other minor bug fixes.

I looked more closely at the actual messages on the newsgroup server and message 3 was a reply to message 2, however, it had no refs in the header. So an order of 2 is correct, but the orphaned replies code only found one reference to associate with it.

Talking about threads, my usenet threads count is always 0 on main page since i have installed 2.3. Post count is correct but thread count doesn't progress.

Also on another note.. user group permissions to post.
- With usenet forum set to 'Open for new posts?' (No) noone can post including user groups allowed to do so.
- With usenet forum set to 'Open for new posts?' (Yes)
everyone can post including canpostusenet = 0 groups BUT only canpostusenet = 1 group propagates to usenet. Confusing bit is that some replies get propagated to usenet, and then canpostusenet = 0 replies appear only on the board. In effect to NNTP users my guys appear to be replying to phantom posts in the thread.

I went through the whole lot few times, can't spot the bug...

Check you have entered the variable name correctly in the templates. It should be '$totalusenetthreads'
That's exaclty how it's supposed to work. This way there is no interference with the vB forum permissions. All the canpostusenet flag does is give you one more thing to play with when you're setting up forum permissions. It will not stop anyone posting to a forum (the vB permissions are already available for you to do that). As you have correctly determined, all it does is enable/disable propogation to 'usenet'. I will clarify that in the documentation of the next release.

Updated version - New features & bugfixes

This release adds a few options and fixes some of the bugs discovered over the last few days.

• Added option in control panel for eliminating usenet users and posts from the stats.
• Added option in control panel for eliminating usenet posts from the 'Get New Posts' search.
• Added option in control panel to allow multiple iterations of replies load per batch run to avoid staggling posts left in the article table when pulling history or large batches.
• Removed hardcoded regexes that were removing things like 'fred wrote in message <123213@news.news.com>'. This should now be entered in your 'Replacements' in the 'Spam Control' section of the control panel.
• Corrected usenet user post count when importing usenet users
• Corrected miscounting of replies in the forum table. (this was confusing vB and causing the next thread link to display when it shouldn't)
• Disabled the refresh in the control panel after updates to prevent the 'cant send header' errors when gzip or buffered output is disabled.
• Prevented orhan replies with the subject of '(no subject)' being threaded incorrectly alongside non-related articles with no subject. They are now not loaded at all unless there is a reference in the header.

Download version 2.4 for vB 2.0b3

There are upgrade instructions for anyone who has installed v2.3. There are only two places in which to change code and a few SQL statements to run. I'll be around for a few hours if anyone installs it tonight and runs into problems. But it's straightforward enough... trust me :)

Both the newnews.pl and the usenet.php need replacing with this new version

The latest version of this hack will always be in this first post of the thread.

Fastworward, help!

A user asked me about some crap messages shown in a non-usenet forum, lots usenet posts that arenet from the one I pull with things like:

"This message has been automatically sent by Micromuse's Netcool/NNTP
Internet Service Monitor."

and some SPAM etc. dated Feb 12, but added today.

The forum in question is

http://www.lowcarber.org/forum/forum...p?s=&forumid=2

I'm still using 2.3. Is this a new bug, or a known one fixed in 2.4?

I only use one usenet group as forum #30 and it is populated correcly, while this stuff shows in forum #2.

Don't panic! The only way this can happen is if you have a destination forum for one of your usenet groups set to a real forum.

Just look in your usenet groups option in the control panel and find the offending group. If you're getting automated messages it sounds like it's alt.test or another test group.

If you're absolutely sure it's not showing on your groups options page then look in the usenet_group table for it. But there is no way this can happen by itself. The script has to be told which forum to place messages in.

Hmm,

I've deleted alt.test (was it set by default to forum #2)? I might've had run it by mistake before I set up the only usenet group I carry. My usenet server has a huge retention rate, so feb12 messages soesn't surprise me.

Nope. It was set to forumid 9999. This was done to avoid the very problem you've just ran into. Maybe I'll just leave the test group out altogether next time.

But if the messages have just appeared today, the script must still be pulling them from usenet. Was it in the usenet_group table?

No, my database shows one group only, alt.support.diet.low-carb, and the same in my CP.

I might've just noticed it today when a user reported it, since the retention is hight on y NNTP server, I pulled feb 12 messages in late March, so they appear at the bottom and don't get noticed. That's what comes to mind.

Hmmm, sorry about that. I think I will leave the alt.test group out of the next release... just to be safe. But there really is no way for the script to place messages in any forum other than that specified in the usenet_group table.

Can you clean up the forum easily? It's easy to identify a post as usenet, but it's not easy to determine which forum it's in because there's no forumid in the post table. If there's not too many it's probably safer to do it by hand rather than a query.

Let me know if you need any help.

Oh it could've been a screw up on my part. Remember my earlier post when I was using the 2.1 script on top the 2.3 version of the hack.

Thanks for the offer of help. There are quite a few of them to do by hand.

How can I identify usenet messages. I might be able to do something like

"delete all usenet posts where forum id=2"

If that what you ment by the difficulty due to not having forum id in post table, I can try this:

"delete all usenet posts prior to feb 13"?

There is a column in both the post table and the thread table called 'isusenetpost'. The threads are easy to delete as there is also a 'foumid' column there. But there is no way of knowing which usenet articles from the post table are from the forum that needs cleaning as there is no forumid column.

The cleanest way may be to purge all usenet posts and re-pull the news. You can opt to keep articles that are usenet, but were posted by your users in the control panel if that is a requirement.

I have no problem purging all usenet posts and polling them back again. How can I purge them? Just to make sure I got it right:

Delete all threads where is usenetpost=1
Delete all posts where isusentpost=1

Any other queries I need to do?

You can do it from the control panel in the 'Groups Section'

There is an option at the very top to purge all posts, or you can do it for each group.

But if you want to do it by hand you are correct in what you said. 'where isusenetpost=1'

Oops, that didn't work right. It just deleted the posts from the usenet forum, not from the screwed up one! I'll table by table.

Ah.. yeah.. sorry. I should have realized that. It's another safety feature to only delete from forums that are specified in the usenet_group table. As you've deleted it from that table it ignores it.

Fastforward, I think I found the explanation to what happened. I grep'd on test and found this:

vB_sql_changes.sql:INSERT INTO usenet_group (newsgroup, forum, lastmsg, server,
username, password, enabled, footer) VALUES ( 'alt.test', '2', '0', '', '', '',
'1', '');

I'm not sure if this was 2.1 or 2.3, but one of them probably wasn't 9999. Just in case anyone has the same installation.

Those funny threads were probably sucked in before I deleted alt.test.

Everything's fine now, and I'll install the 2.4

The link for downloading 2.4 doesn't work?

Doesn't work for me, but it did earlier. Here's a link that works: http://britishexpats.com/download/us...4__20b3.tar.gz

Well, I don't kow what to say, except I'm very sorry. I'm guessing I overwrote my original statement that had 9999 when I gave the docs an overhaul. To ensure everything worked I extracted the DDL & DML from my setup using phpMyAdmin. My setup was just for test and I did have the alt.test group pointing to forum #2.

I have now removed the alt.test group completely. I'll also add a note in the docs to double check the groups options page and make sure the forums specified are really where you want the usenet messages to go before runing the script for the first time.

The download link was a good one, but I woke up this morning to find my disk was full. Apparently I no longer have access to /dev/null so all my log files filled up. I guess I'll be on the phone all day to Host Pro :rolleyes:

Has anybody installed 2.4 yet? Is it all working OK?

No problem Fastforward, was just a minor inconvenience, nothing compared to the work you've done on this hack, which is the best vb hack in my book:)

With 2.4 i'm actually getting
Parse error: parse error in /usr/local/apache/htdocs/forum/admin/functions.php on line 871

Fatal error: Call to undefined function: vbdate() in /usr/local/apache/htdocs/forum/admin/sessions.php on line 326

in both admin/ and forum

It's the
$DB_site->query("UPDATE forum SET lastactivethread = LEFT('"..addslashes(htmlspecialchars($threadinfo[title]))."',$lastactivethread_length) WHERE forumid = ".$threadinfo[forumid]);
line...

I wrote a script that let's you insert an archived newsgroup or mailing list into the usenet tables. I wrote it so that I could import all the messages since the newsgroup was created and so that I could feed the new messages in from two sources, the mailing list and the newsgroup. Feeding it in from the two sources, I hope will eliminate my missing posts problem for the ones that don't make it into the newsgroup.

This script will read mail in from a file or from an email alias and then parses it to be included in the database. Then the messages will be inserted with the newnews.pl script provided in the Usenet Gateway hack. This script does not make use of any of the spam/binary filtering or other replacement strings that occur before it's inserted.

How to use this script:

From the command line, execute the script like:
That will insert all the messages in the text file into the database. The text file needs to conform to the mail standards, so it's the same format as the sendmail spool files and the same as what many other email programs use. (Basicly, the beginning of a message is signified by a line beginning with "From " followed by the headers, then an empty line, and then the body).

This script can also be used to insert a message sent to an email address right into the database. To do this, you'll need to create an alias in the sendmail aliases (usually at /etc/aliases) to execute the script. The alias would look like this:
Most systems are set up so that not just any script can be executed by sendmail for security reasons. Usually, you'll need to create a symbolic link to the file in a special directory (as root). In the systems I've used, this directory has been "/etc/smrsh/" or "/usr/adm/sm.bin/".

You'll need to rename the file to have the .pl extension.