You have to create a forum using the vB functions for every usenet group you want. Then you also have to enter the newsgroup and assign it to the vB forum using the usenet control panel page.

If you did that and they're still not showing, make sure the messages you pulled are current and not just hidden from view by the forum settings :)

Be careful that you assign the correct forum to the newsgroup or you may end up with usenet messages in a real forum!

Fastforward,

Hmm - I commented out that line, there seems to be a slight increase in speed but not really anything significant.

I've been watch what is going on with my server and i'm wondering if it is something to do with the way mysql is setup on my system...as i've said i'm running it on a Cobalt RAQ4i that people in these forums seem to have mixed feelings about...whilst studying what is going on with the mysql processing during the long periods the following processes seem to spend a long time:

The state it seems to be in a lot is:

Which leads me to believe that mysql isn't perfoming well. This is strange as I have 512 Mb of RAM with 30 GB HDD - I followed instructions on changing some of the sartup parameters for mysql but nothing really seems to have a great impact.

Any suggestions?

That query probably will use a fair bit memory as it's joining to the post table. This was added to eliminate the need for a unique constraint on the msgid column in the post table. The unique key had a potential to cause problems with vB upgrades and uninstalling the hack etc.

You can remove the DISTINCT clause from the query. That is causing a sort when it's not really required as we're joing on msgid which should be unique in both tables. See if that stops it creating a temp table.

Also, make sure you have indexes on the msgid column in the post table and the msgid column in the usenet_article table.

The usenet_article msgid should be the PRIMARY_KEY or UNIQUE.

The post msgid should be a non-unique index.

Has the performance always been the same?
If that still doesn't work, you might try removing the LOW_PRIORITY from the statement. It shouldn't make any difference, but you never know.

OK - I think I hit gold here...I double checked the indexes on those tables and found the post msgid index missing...I added that and performance has increase by about 1000%! Look like that is the major problem - must have missed that during an upgrade.

I didn't do any other changes...i'm also going to reinstate the line you recommended to comment out of newnews.pl.

I just noticed the Spam Control page in the control panel isn't refreshing properly after the cosmetic changes made.

I've put a revised usenet.php in the package and made the version 2.8.1.

Only the usenet.php changed so you can just download this one file here and overwrite yours.

The code in here is a total mess and really needs sorting out. I still get the 'headers already sent error' ob the spam page, but it works. I'll look at tidying it up later (when I learn some more PHP :) )

I would really like to be able to use this hack on my forums but I'm concerned that it would eat through the bandwidth (data transfer) on my site and cost me a lot of money.

Does anyone who has used it have any figures about how much bandwidth it would use, obviously it's different for different newsgroups but can anyone give me a general idea?

If I could put it on my site and it would only take (say) 1 or 2GB per month then that would be great but if it was to take (say) 20GB that would be impossible for me.

Thanks

Mark,

Well, I haven't got any firm figures but here are my general thoughts on this:

- First reaction is not much!
- Biggest drain will be when you do your first batch pull as this is getting a lot of posts for history
- Depends on how many newsgroups you pull...
- ...and how active they are
- Since the perl script ignores Binary Posts (or posts above a certain amount of lines) these are not a drain on your bandwidth

Overall I don't think it should be too bad, but i'm sure others have some sort of analysis on this.

i would have thought hardly anything if you are only pulling a text database. All it you are doing is pulling some newsposts and getting the article index, i would be suprised if you get close to 100MB in a month mirroring 5 or so reasonably active text news groups.

Really! I was expecting much more than that!. Outstanding news if it's accurate. And yes I was thinking about doing about 5 text newsgroups.

PS Does this hack work ok on BETA5? (sorry if thats already covered)

it is only an estimation but if you think about it what other data is there going to be? all the data you are pulling is the actual text posts and they go straight into your db. If you were pulling 1GB of data a month the worry would not be the bandwidth problem but the size of your database! I think matt said recently at sitepoint that a 1,300,000 post database was about 1.5GB or something which should give you an indication... no way will you pull anything near like a GB per month i am pretty certain of it.

About 1 more week i recon until i have another go with this, it has changed a lot since i first installed it on v 1.5 ;)

It almost works on beta 5

You may have problems with the usenet.php page that generates the control panel as that changed in RC1. Also the line numbers in the instructions for code changes will ahve changed in many places.

What you can do is download this old version for beta 5 but use the newnews.pl from the latest version.

NOTE: It's imperitive that you DON'T use the old newnews.pl script from the beta 5 package as it has a serious bug in the auto-purge department! (ask Tamarian... he'll tell you!). Everything except newnews.pl should be used from the old package for beta 5.

As for bandwidth, Chris is right. You can use the the size of the database as a rough guide. I haven't actually looked, but my vserver account only allows 20GB/month and I have 2 sites with 75,000 page views pulling about 80 newsgroups between them and I haven't had a problem. You'll be using megabytes rather than gigabytes.

I am using multiple servers and posts to one of the servers does not make it to the newsgroup. Looking at the output of running the newnews.pl script, all outgoing posts are trying to be posted on the first newsgroup it processes which for posts that are going to a newsgroup that is not on that server, it means it doesn't make it to the newsgroup but says it was sent. Disabling all the newgroups on one server makes it where the posts go to the right server (but tries to send the messages for the disabled newsgroups too). It does the same for inserting replies where all of them are done on the first newsgroup, but they go to the correct forum, so that's not a big deal.

This has happened since I upgraded to the vb2.0b5 version and I just tried it with the newest newnews.pl script and it occurs with this one too.

I have not investigated further... I would assume it's a coding problem or a problem with the database tables (something not upgraded properly). Does this happen for others that are using multiple servers?

Here is the output from newnews that I get:
Connecting to news.tc.umn.edu... Connected
Sending authentication info... Authenticated and logged in
Getting article batch from rec.sport.unicycling
No new messages in rec.sport.unicycling
inserting new threads into forums
finding replies...
checking for orphans...
Processing outgoing messages
Posting message by Gilby to webdiscuss.test... Sent

You can see that it is trying to post to "webdiscuss.test" while processing for "rec.sport.unicycling". The "webdiscuss.test" newsgroup is disabled and also is on another server. This post did not make it to the newsgroup even though it says "Sent".

I've had a quick look and can't see anything obvious, so I'll need to spend some time trying to find out what's wrong. I should have something by tonight.

One point to note though. The sent message only means the server accepted the message. The server will not check to see if it's a valid group first. That validation should really be handled by the script. However, as the groups are pre-defined within vB the chance of sending to an invalid group are practically zero... except in this case :rolleyes:

I think its due to misplaced parenthesis in the WHERE clause of the post_outgoing() function. As it is at the moment, it will:

either (get messages with no refs) OR (get aanything with a subject that doesn't start with Re: and is in this group)
Therefore it will match any message that has no refs. regardless of whether it's in the right group.

What it should be doing is:

(get messages with no refs or ones that have a subject that doesn't start with Re:) AND (also make sure they are in the newsgroup that is currently being processed)

Here's the fix:

Replace line 750 which currently reads:

my $q2 = db_fetch("SELECT poster,newsgroup,subject,refs,body,threadid,postid FROM usenet_outgoing WHERE (refs <> ' ') OR subject NOT LIKE 'Re: %' AND newsgroup = ". $dbh->quote($$newsgroup->{'newsgroup'}));

With this line:
my $q2 = db_fetch("SELECT poster,newsgroup,subject,refs,body,threadid,postid FROM usenet_outgoing WHERE (refs <> ' ' OR subject NOT LIKE 'Re: %') AND newsgroup = ". $dbh->quote($$newsgroup->{'newsgroup'}));

Let me nkow if that fixes it. I can't actually test it at the moment coz I'm busy at work. :p

The fix mentioned above is now packaged up and the release number is 2.8.2. There was also another little problem with the logic that handled re-importing of forum generated posts that didn't exist in the forum. Unless you had the option to re-import turned on, you wouldn't have noticed this.

Works great now! Thanks.

<problem solved post removed>

When upgrading the board I need to be able to take backups of the database in case things go wrong.

I tried this on moving to a test database but got this
This is a serious error because it means I can no longer take backups of my database :(

Can anyone tell me a fix or point me to the instructions as to how to fully uninstall the hack from the database? I've got the empty_usenet.pl script but want to make sure it will run with the latest version of the hack, since I can't backup my database :(

What exactly are you trying to do? What code generated that error? I can still take backups of my database.

To uninstall the hack, simply upload all your original vB files and drop the columns and table that were added to the vB tables. You can see what these were in the installation scripts.

This release affect newnews.pl only. It fixes a few minor errors.[list=1][*]Problem with email notification not using the vB templates correctly.[*]Fixed autopurge option to correctly clean out the searchindex table.[*]Added additional checks to all queries that delete or update data to ensure that only usenet groups are affected.[list]
You can just overwrite your newnews.pl with the attached file. The new file is also in the full package.

The next release will have a lot of changes that eliminate the need for many of the code edits to vB files. I have already got it working without the need to edit newreply and newthread.php which also means move,copy,split etc will work without problems for any thread.

The other major change is the removal of the option to import usenet users. I think this is a waste of time and space and serves no real purpose except the ability to search by username. For the extra code edits and problems that this introduces, I don't think it's worth it. Anybody have a problem with that?

My aim is to limit the code changes to a state that allows vB upgrades to happen with no loss in functionality to either vB or the hack (except maybe some inevitable display issues such as count of usenet posts etc)

Can't wait for this! :)

[QUOTE]Originally posted by fastforward
What exactly are you trying to do? What code generated that error? I can still take backups of my database.
[quote]

Just dumping the database and attempting to reupload it to an empty database.

Thanks, I'll do that.

I did notice one bug which you may or may not be aware of, when I tried to 'update counters' for last posters etc, it threw a database error when it found a usenet post with a / or a ' character.

I can't wait until your next version of this hack :cool:

If you make a version of this that can run without editing any vbfiles (other than templates) i will bow to you.

I really want to use it but I refuse to edit vb files so when something goes wrong their the only ones to blame :)

Cant wait for the next version :)

This is a first release of the new newnews.pl lite :)

RC2 came out a bit sooner than expected, so I rushed out an early version.

The good news:

• Only one vB code edit to make! And that is just to add a link for the control panel on the admin/index page.
• Discovered and fixed some more bugs and efficiency issues
• Reduced the number of additional columns added to vB tables

The bad news:

1. You have to work with your templates to hide unnecessary fields such as post counts and location etc. As there are no conditional statements or constructs alowed in templates you will have to create a new styleset and assign it to the usenet forums. This could actually be seen as an advantage.
2. Due to the unexpected release of RC2, this release doesn't provide a way to display a seperate count of usenet posts on the front page. Neither does it allow posts to be hidden from the 'new posts' search. I will try to figure a way to do this without editing the code.
3. Finally, there is no extra permission flag to prevent posts propogating to usenet. This option was a little pointless anyway and could be confusing. If you don't want people posting to usenet then just don't let them post in the usenet forum. Use the vB permissions for it.

More good news: I believe it's possible to have a fully featured usenet gateway with zero code edits to vB files.

Once again, this release was a little rushed. I believe the posting and threading are working correctly, but there may be some other issues I've overlooked so use it with care. It shouldn't really affect any of the vB functionality as we don't touch the code anymore.

One more thing, as mentioned in an earlier post, the 'import user option' is gone.

The package has been updated in the first post of this thread or you can get it here.

If you want to continue using the old one with all the code edits, you can still get it by calling up a directory listing at http://britishexpats.com/download and picking the version you want.

Since this doesn't edit vbulletin code anymore is the last thread title "bonus" still included (I assume not).

I use this on my board, is there anywhere I can get instructions as to the code changes needed to implement this on it's own?

Ah.. I didn't think anyone was using this. I'l get the instructions out later today. It's quite easy and only needs editing in one place (indexpost()) if I remember correctly.

Fastforward, great job. I just looked up the instructions. This is virtually no code change? If I use phpMyAdmin to enter my news server info and options, or if the settings are entered in newnews.pl, no code changes are necessary?

Yep. That's right. I did actually have a go at changing the config page to work standalone which would mean you wouldn't even have to make the one little change left, but it was getting late and I broke it :) I'll give it another go if I get time before RC3 or vB final is released.

I've figured an easy way to get the usenet post counts and any other config setting you make in the usenet control panel into a global variable that can be referenced in templates. I can just append them to the options template during each news pull. The only drawback to that is if you update the vB config using in between news pulls, the usenet variables will be overwritten until the next news pull set them again. It won't be a problem as long as you remember to resubmit the usenet config straight after you update the vB config. And if you really can't remember to do that, it would only be one little optional code change to functions.php to automate it. I'll try to find another way, but that method is favourite at the moment.

I still haven't figured out a way to eliminate the usenet posts from 'new post' searches without code edits. I'm sure there must be a way though :)

I've tried the new version but all the posts have the date/time of when they were inserted into the database rather than the correct date when they were posted.:(

That is how it's supposed to work. Otherwise, they won't be threaded correctly when local posts are in the same thread as usenet posts.

If you want it the old way then you will have to go back to the old method of editing the vB code in newreply.php, newthread.php, postings.php and functions.php. Which would you prefer?

The time of posting is not very reliable anyway. It's not even the NNTP time. It's based on the users PC from which the post was made. With this method, the post times are actually going to be more consistent. Plus. if you're pulling news 2 or three times an hour, there won't be much discrepency between the original post (except when the original post was wrong).

I like the old way better, where the date of the post is the same as the date in the message on the newsgroup. Maybe some date checking would be appropriate such as making sure the date isn't in the future and that it's not older than a few hours or since the last time newnews was run, whichever is older.

Let's say that either the news server or your own server was down for a while. You don't want to have the messages that were not loaded earlier being the same date as all the others. Sometimes it takes a while for messages to make it to the a particular news server and they might come out of order too.

With this method, posts will appear just as they do using a newsclient against a newserver. The news server simply displays them as it receives them.

If the date of the news post is used and placed into the dateline field, that is how vB orders the posts. This means they will NOT be shown in the correct order due to the time taken for news to propogate across usenet. You will get usenet posts coming in that are placed in the middle of threads. Also, they may not show up as new posts as they may be dated before your last visit. It becomes impossible to keep track of long threads.

The other problem is when a local reply is posted in a usenet thread. You will also get a whole bunch of usenet posts placed above it when the next run occurs. The local poster gets an email telling them someone has replied and can't see any posts after their one. Very confusing and frustrating.

These example aren't isolated occurances. It happens all the time. In fact, the very first version of this hack did exactly what you are asking for. It was changed for the reasons above.

The way it was handled in the more recent versions was by using the 'seq' (sequence) field. The thread was orderd using this column which meant everything was shown in the right sequence (although the message dates were not necessarily in order). This was achieved by editing the showthread, newreply, newthread and posting.php.

As far as checking goes, this was also done in the previous version to minimize the message dates appearing strange and out of sequence, and also to make sure posts were shown as new in a timely fashion. But, it's only of use when combined with the vB code changes.

Basically, you have a 3 choices.

• Use the news post date with no vB code changes and have posts that you may never see as they will be inserted in the middle of threads and not be marked as new.
• Use the method of setting the date to the date they are inserted into the forums. (Just as a real posts do). The posts are in the correct order, but have a 'wrong' date.
• Use the newspost date along with all the code changes that are necessary to support it and the inability to upgrade to newer vB versions without breaking the hack, vB, or both.

trust me, you want to stick with this way! getting posts going above native VB posts is incredibly confusing and as fastfoward says it happens all the time. Newsgroups themselves don't sort by date/time, VB shouldn't do either for newsgroup posts and threads.

That is actually how I would like it to work, would it be possible to set this as an option in the usenet.php control panel as I really don't like the way it is done now.


OK, that "bug" now found to be a "feature" :). I have another one for you :rolleyes: !

The names don't appear in the individual posts. Even though they do appear in the thread starter and last poster columns on forumdisplay.

Also I don't know if this is to do with your code but threading on outgoing posts doesn't seem to be working correctly. There was a long thread with 5 replies and I clicked reply and replied to the message. I expected the post to be in reply to post number 5 instead it appeared in reply to the first post.

This isn't possible without some major code changes. vB is not a threaded bulletin board. If vB doesn't know or care which post is being replied to, it's impossible for an external script to figure it out.

It would be possible to figure out if an individual post was being replied to by using the quote feature, but that would mean more vB code changes and it would only work if the quote was used.

This hack has always only 'threaded' incoming messages by placing them in the correct sequence and a logical order. Outgoing messages have always been in reply to the first post.

As for the name not appearing, I'll have to look into that some more. They seem to be showing up on mine.

Ah right, that's a pity because it will look quite strange on usenet. (There quite a fickle lot over on the newsgroups). Can't newreply.php be modified to include a "in reply to" tag which would handle this. (code changes aren't that bad really!).

Thanks, see example here -
http://www.motorsportforum.com/vb22/...=&threadid=235

It could. But the whole point of this 'new' usenet hack is to eliminate all code changes to ease support issues for everyone and allow timely upgrades to new vB versions. It also means I don't have to stay up until 4am the day Jelsoft releases a new version :)

Remember this new version is still in it's infancy after stripping out all the code changes. A lot of the missing features will be readded... and still with no code changes. I just need time to figure out how.

By the way, I can add an option for the usenet post date issue. But remember; new posts may be placed before recent posts. The dates will obviously be in sequence, but you may receive the newer post first. The real problem is when local posts are entered into a thread along with usenet posts. You will nearly always get usenet posts placed before it. I'll add the option with a disclaimer.

Fastforward,

Here's a suggestion. No vb code changes, and minimal usenet customization can be the basic usenet hack. A list of mods to the hack can be updated, by you and other experienced users of the hack. Like a text in the hack zip with a format like:

1. Search: To exclude usenet posts from being displayed in view new, find this in search.php, replace with, etc.

2. Stats to eclude from stats, do this...

bla bla. Each needed enhancement can be hacked by itself, while the simplest usenet hack won't need any code changes.

I can see this hack taking a lide of it's own!

I would be in favour of that but I doubt fastforward would since he is doing this for free after all!

However another thing I found was that wereas previously the usenet posts didn't appear in my overall post total they do now, this is bad.

We probably won't go through another repetitive upgrade cycle like vb1.6-vb2.0 for a while. Some mods to the Usenet hack are quite simple, especially if you hard code them instead of using the CP. Like adding "AND forumid <>" in search to control search.

I actually use the 2.8.2 of the hack on top of RC2, so it still behaves like it used to, and will install 2.9 when vb2 is released, with mods for stats msg count and seach behaviour on the usenet messages.