Usenet gateway

[fastforward]

Originally posted by TechTalk
Every time i have tried to view the examples or download the file, both http://britishexpats.com and http://www.dbforums.com are down :(
eh? When?

According to my internetseer reports, BritishExpats.com has been down once over the past 2 months sometime between 9am and 10am a few days ago.

How often have you tried? Where from and when?

[chrispadfield]

at this rate you are getting very close to running out of possible features to add :)

[TechTalk]

Im trying both as we speak.

C:\>tracert BritishExpats.com
Unable to resolve target system name BritishExpats.com.

Thats a trace from 24.217.12.87 (home)


[cmiller@server1 cmiller]$ /usr/sbin/traceroute BritishExpats.com
traceroute: unknown host BritishExpats.com


From my dedicated box 209.61.188.107 (other end of the country)

[TechTalk]

About 2 mins after i started writing the above post, both sites became available.

~Chris

[Gilby]

Originally posted by chrispadfield
at this rate you are getting very close to running out of possible features to add :)

There are always new features to be added.

Here are a couple that are on the top of my list:
Support for multi part messages, so only the text part is inserted into the post, and the html version is tossed.
Attachments, if from a newsgroup, then it'll make it a regular attachment in vB. If posted on the forums, a link at the end of the newgroup post is given for where people can download it.
Better text formatting int he newsgroup. (Possibly take the post, and parse the vB code into html and then use the html to text perl module). Good for stuff like lists.

[fastforward]

Originally posted by Gilby
Better text formatting int he newsgroup. (Possibly take the post, and parse the vB code into html and then use the html to text perl module). Good for stuff like lists.
[/list]
I take it you mean the outgoing posts? This is something I haven't really spent much time on yet. But I agree, it's something that needs improving. I take the easy option at the moment and disable BB code & html for outgoing posts in the forum :)

As for incoming posts; there's not much else we can do with that. The posts that look bad in the forums look just as bad on usenet due to crap news clients or people posting using MS Outlook. I'm sure it wouldn't be too hard to handle the dual format messages correctly though. I'll get on it :)

[Gilby]

Originally posted by fastforward

I take it you mean the outgoing posts? This is something I haven't really spent much time on yet. But I agree, it's something that needs improving. I take the easy option at the moment and disable BB code & html for outgoing posts in the forum :)

Yup, I think it'd be cool if the posts that come out of the forum would have good formatting and are better than any other formatting of plain text documents.


As for incoming posts; there's not much else we can do with that. The posts that look bad in the forums look just as bad on usenet due to crap news clients or people posting using MS Outlook. I'm sure it wouldn't be too hard to handle the dual format messages correctly though. I'll get on it :)

Most the html posts that make it on my forums are from AOL users, now that AOL has AOL 6.0 out, they now send html formatted posts instead of plain text out to non-AOL users by default. Of course there is the occasional html post from other people as well.

[tamarian]

Hi Fastforward,

When I switched autopurge on, I got an error from newnews.pl, for invalid isreg field?

Looking up the ugrade sql statements, this field was not added. Shoul it have been there, or is it from previous versions?

[fastforward]

Sorry. my fault. It's the same in mine but I didn't notice :)

You need to change
isreg
in newnews.pl to
regpost

There is only one reference to it on line 856.

It's referring to a column in the thread table. If it's set to 1, it means the thread has been posted in be a registered member. It gets updated in newthread.php and newreply.php. Those files seem to be OK though, so you only need to change the one occurence in newnews.pl.

[tamarian]

Now it purges fine, thanks.

[fastforward]

DO NOT turn on autopuge. Tamarian has uncovered a serious bug that purges non-usenet groups aswell as the usenet groups. This is due to a left join being used to get the forumids.

The fix will be released ASAP.

[fastforward]

The auto-purge bug is now fixed. It will now wait until all news has been collected and then purge ONLY groups that are in the usenet_group table. The links have been updated to point to the new version 2.7.2.

To upgrade, you only need to replace the newnews.pl script.

[Stephan Whelan]

Just installed 2.7.2 and have to say that it all seems to work great...auto-purge as well.

Congratulations again fastforward!

:cool:

[TechTalk]

You dont know how bad this is needed in vb 2.1!

[Stephan Whelan]

fastforward,

well it looks like I was too quick in saying that the auto-purge worked.

I am running v2.7.2 of your hack but it seems that the auto-purge still doesn't work the way it's supposed to.

It is currently deleting anything older than 28 days(what I set in the USENET settings) from all forums - not just the usenet ones.

[fastforward]

It's working on mine.

Check the article_purge function in newnews.pl. It's at the bottom starting at line 851.

It should like exactly like this:

sub article_purge {
my $forums = db_fetch("SELECT forum, newsgroup FROM usenet_group");
while (my($forumid,$newsgroup) = $forums->fetchrow_array) {
my $purge = time() - ($config{auto_purge_days} * 86400);
console("Purging usenet threads older than $config{auto_purge_days} days from $newsgroup...");
my $threads = db_fetch("SELECT threadid FROM thread WHERE forumid=$forumid AND lastpost < $purge".(!$config{auto_purge_reg}?" AND regpost = 0":""));
while (my($threadid) = $threads->fetchrow_array) {
my $posts = db_fetch("SELECT postid FROM post WHERE threadid = $threadid");
while (my($postid) = $posts->fetchrow_array) {
db_execute("DELETE LOW_PRIORITY FROM searchindex WHERE postid = $postid");
}
db_execute("DELETE LOW_PRIORITY FROM post WHERE threadid=$threadid");
}
db_execute("DELETE LOW_PRIORITY FROM thread WHERE forumid = $forumid AND lastpost < $purge");
console(" OK\n");
}
console("Purge completed.\n");
}

[Stephan Whelan]

Here is the section in my newnews.pl...doesn't seem to be exactly the same...


sub article_purge {
my $forums = db_fetch("SELECT forum, newsgroup FROM usenet_group");
while (my($forumid,$newsgroup) = $forums->fetchrow_array) {
my $purge = time() - ($config{auto_purge_days} * 86400);
console("Purging usenet threads older than $config{auto_purge_days} days from $newsgroup...");
my $threads = db_fetch("SELECT threadid FROM thread WHERE forumid=$forumid AND lastpost < $purge".(!$config{auto_purge_reg}?" AND regpost = 0":""));
while (my($threadid) = $threads->fetchrow_array) {
my $posts = db_fetch("SELECT postid FROM post WHERE threadid = $threadid");
while (my($postid) = $posts->fetchrow_array) {
db_execute("DELETE LOW_PRIORITY FROM searchindex WHERE postid = $postid");
}
db_execute("DELETE LOW_PRIORITY FROM post WHERE threadid=$threadid");
db_execute("DELETE LOW_PRIORITY FROM thread WHERE forumid = $forumid AND lastpost < $purge");
}
console(" OK\n");
}
console("Purge completed.\n");


I'm going to change it to your code.

[fastforward]

You must have downloaded that within seconds of me uploading the version!

I replaced it within 2 or 3 minutes of initial upload.

Sorry about that. I didn't think anyone had got to it first. Did you lose any posts?

[fastforward]

This release just updates things for vB 2.0 RC1. There were very few changes made. The main changes were the control panel, to make it look like the new vB style, and the line numbers in the documentation.

Download (http://britishexpats.com/download/usenet_gateway.tar.gz) latest version for vB 2.0 RC1

The latest version of this hack will always be in this first post of the thread.

[Stephan Whelan]

fastforward,

OK - it now is working fine - that will teach me to be too quick at downloading updates!! I didn't lose too much as we only changed to vB from UBB in the last month so just re-imported old posts from the old forums.

Anyway, I do have one question though (I feel you'll learn to hate me:D ) :

- How long does it take you to process the newsfeeds. I have 12 newsgroups running and I have to set my cron schedule to every 2 hours (I might put it back to every 3 hours) as it takes on average around 80 minutes to complete a run at processing all posts. I have around 500 posts in the usenet_article table and the expiry length is set to 2 days in my CP. I'm not to fussed in some respects, but I am a bit concerned that there is almost a constant load on the server...

What are other peoples experiences with this?

[fastforward]

Originally posted by Stephan Whelan
fastforward,
- How long does it take you to process the newsfeeds. I have 12 newsgroups running and I have to set my cron schedule to every 2 hours (I might put it back to every 3 hours) as it takes on average around 80 minutes to complete a run at processing all posts. I have around 500 posts in the usenet_article table and the expiry length is set to 2 days in my CP. I'm not to fussed in some respects, but I am a bit concerned that there is almost a constant load on the server...

Woooah!!

You have a problem! I pull, process and load 70 groups in 5 minutes! When it was on a shared virtual server it used to take up to 15 minutes.

Even when pulling history with batch limit set to 150 it only takes 10-20 minutes! That's over 10,000 posts.

What version of Perl are you using?

[Stephan Whelan]

fastforward,

Perl is 5.005_03.

I've watched the server pull newsgroups stuff through and it pulls the articles very quickly, the problem seems to be when it reaches "Checking for Orphans" and "Loading x Replies" stage.

I've followed details on optimising MySQL from various threads in these forums but doesn't seem to help very much.

It isn't a shared server, it is a cobalt raq 4i with 512 RAM.

[c0bra]

Since trying to apply this hack our forums have been down with the following error:

Parse error: parse error in /home/sites/site3/web/forums/admin/functions.php on line 1729

Fatal error: Call to undefined function: vbsetcookie() in /home/sites/site3/web/forums/admin/sessions.php on line 198

I followed the directions exactly as specified in the INSTALL.TXT file. Modified the required files and then manually uploaded them to the server.

Any ideas on the problem? A prompt solution would be much appreciated since our boards are down at this moment. :eek:

[chrispadfield]

first- replace functions.php with the backup you made to get forums working again.

then look inside functions.php the error might be beginning here:

function vbsetcookie ...

basically you have done something wrong in fuctions.php, missed of a closing bracket would be my guess

[c0bra]

I uploaded the original functions.php and indeed the forum started working again. I then proceeded to go through the process of modifying the functions.php file again double and triple checking eachtime that the changes I made were absolutely correct. However, when I uploaded the file again the error I posted above immediately reappeared.

Maybe someone could attach a copy of their working modified functions.php file so I can take a look?

[chrispadfield]

i havn't done it yet so can't help.

You are using the correct version of vb for the hack, release candidate 1. I could not tell from your forums as the version number is not there.

[fastforward]

Originally posted by Stephan Whelan
fastforward,
Perl is 5.005_03.
I believe this may be your problem. The email::find module requires version 5.005_63 to work at it's optimum. Although having said that I was using it with and older version with no problems. Here is what the module author states:

This module requires 5.005_63 or higher!
This module runs so slow as to be unusable with 5.005 stable. I'm not sure, but it might be because I build up my search regex using lots of compiled regexes. Either way, it runs orders of magnitude faster under 5.005_63.

To confirm my suspicion, try commenting out line 828 of newnews.pl.
It currently reads

find_emails($from, sub { my($e,$orig) = @_; $email = $e->format; } );

It won't make any difference to the functionality unless you are importing usenet users.

Let me know what happens.

[fastforward]

Originally posted by c0bra
Since trying to apply this hack our forums have been down with the following error:

Parse error: parse error in /home/sites/site3/web/forums/admin/functions.php on line 1729

Fatal error: Call to undefined function: vbsetcookie() in /home/sites/site3/web/forums/admin/sessions.php on line 198

This indicates there is a brace, semi colon or some other syntactical error. I'll have a look at my instructions again, although it worked when I did mine using my own instructions.

[fastforward]

I've checked the instructions and it works for me. Not what you wanted to hear but it does :)

You might want to re-check the two edits in the deletethread() and deletepost() functions.

The edits replace the closing brace with another line of code AND a closing brace. Make sure you don't have two closing braces in there.

[c0bra]

Thanks your solution worked for me. :D I created a new category forum called "usenet". Created the groups I wished to display on our VB. Ran the perl script, it connected fine, reported downloading all the messages fine, and disconnected fine. However, when I view our forum homepage the newsgroups are not showing up at all.

Should the newsgroup automatically appear under the chosen category when it's created in admin?

Any ideas?

[fastforward]

You have to create a forum using the vB functions for every usenet group you want. Then you also have to enter the newsgroup and assign it to the vB forum using the usenet control panel page.

If you did that and they're still not showing, make sure the messages you pulled are current and not just hidden from view by the forum settings :)

Be careful that you assign the correct forum to the newsgroup or you may end up with usenet messages in a real forum!

[Stephan Whelan]

Fastforward,


To confirm my suspicion, try commenting out line 828 of newnews.pl.
It currently reads

find_emails($from, sub { my($e,$orig) = @_; $email = $e->format; } );



Hmm - I commented out that line, there seems to be a slight increase in speed but not really anything significant.

I've been watch what is going on with my server and i'm wondering if it is something to do with the way mysql is setup on my system...as i've said i'm running it on a Cobalt RAQ4i that people in these forums seem to have mixed feelings about...whilst studying what is going on with the mysql processing during the long periods the following processes seem to spend a long time:


INSERT LOW_PRIORITY IGNORE INTO usenet_article_loader SELECT DISTINCT a.newsgroup, a.forum, a.msgid


The state it seems to be in a lot is:


Copying to tmp table


Which leads me to believe that mysql isn't perfoming well. This is strange as I have 512 Mb of RAM with 30 GB HDD - I followed instructions on changing some of the sartup parameters for mysql but nothing really seems to have a great impact.

Any suggestions?

[fastforward]

That query probably will use a fair bit memory as it's joining to the post table. This was added to eliminate the need for a unique constraint on the msgid column in the post table. The unique key had a potential to cause problems with vB upgrades and uninstalling the hack etc.

You can remove the DISTINCT clause from the query. That is causing a sort when it's not really required as we're joing on msgid which should be unique in both tables. See if that stops it creating a temp table.

Also, make sure you have indexes on the msgid column in the post table and the msgid column in the usenet_article table.

The usenet_article msgid should be the PRIMARY_KEY or UNIQUE.

The post msgid should be a non-unique index.

Has the performance always been the same?
If that still doesn't work, you might try removing the LOW_PRIORITY from the statement. It shouldn't make any difference, but you never know.

[Stephan Whelan]

Also, make sure you have indexes on the msgid column in the post table and the msgid column in the usenet_article table.

The usenet_article msgid should be the PRIMARY_KEY or UNIQUE.

The post msgid should be a non-unique index.


OK - I think I hit gold here...I double checked the indexes on those tables and found the post msgid index missing...I added that and performance has increase by about 1000%! Look like that is the major problem - must have missed that during an upgrade.

I didn't do any other changes...i'm also going to reinstate the line you recommended to comment out of newnews.pl.

[fastforward]

I just noticed the Spam Control page in the control panel isn't refreshing properly after the cosmetic changes made.

I've put a revised usenet.php in the package and made the version 2.8.1.

Only the usenet.php changed so you can just download this one file here and overwrite yours.

The code in here is a total mess and really needs sorting out. I still get the 'headers already sent error' ob the spam page, but it works. I'll look at tidying it up later (when I learn some more PHP :) )

[Mark Hewitt]

I would really like to be able to use this hack on my forums but I'm concerned that it would eat through the bandwidth (data transfer) on my site and cost me a lot of money.

Does anyone who has used it have any figures about how much bandwidth it would use, obviously it's different for different newsgroups but can anyone give me a general idea?

If I could put it on my site and it would only take (say) 1 or 2GB per month then that would be great but if it was to take (say) 20GB that would be impossible for me.

Thanks

[Stephan Whelan]

Mark,


Does anyone who has used it have any figures about how much bandwidth it would use, obviously it's different for different newsgroups but can anyone give me a general idea?


Well, I haven't got any firm figures but here are my general thoughts on this:

- First reaction is not much!
- Biggest drain will be when you do your first batch pull as this is getting a lot of posts for history
- Depends on how many newsgroups you pull...
- ...and how active they are
- Since the perl script ignores Binary Posts (or posts above a certain amount of lines) these are not a drain on your bandwidth

Overall I don't think it should be too bad, but i'm sure others have some sort of analysis on this.

[chrispadfield]

i would have thought hardly anything if you are only pulling a text database. All it you are doing is pulling some newsposts and getting the article index, i would be suprised if you get close to 100MB in a month mirroring 5 or so reasonably active text news groups.

[Mark Hewitt]

Originally posted by chrispadfield
i would be suprised if you get close to 100MB in a month mirroring 5 or so reasonably active text news groups.

Really! I was expecting much more than that!. Outstanding news if it's accurate. And yes I was thinking about doing about 5 text newsgroups.

PS Does this hack work ok on BETA5? (sorry if thats already covered)

[chrispadfield]

it is only an estimation but if you think about it what other data is there going to be? all the data you are pulling is the actual text posts and they go straight into your db. If you were pulling 1GB of data a month the worry would not be the bandwidth problem but the size of your database! I think matt said recently at sitepoint that a 1,300,000 post database was about 1.5GB or something which should give you an indication... no way will you pull anything near like a GB per month i am pretty certain of it.

About 1 more week i recon until i have another go with this, it has changed a lot since i first installed it on v 1.5 ;)

[fastforward]

Originally posted by Mark Hewitt
PS Does this hack work ok on BETA5? (sorry if thats already covered)
It almost works on beta 5

You may have problems with the usenet.php page that generates the control panel as that changed in RC1. Also the line numbers in the instructions for code changes will ahve changed in many places.

What you can do is download this old version for beta 5 (http://britishexpats.com/download/usenet_gateway_v2.7.2__20b5.tar.gz) but use the newnews.pl from the latest version (http://britishexpats.com/download/usenet_gateway.tar.gz).

NOTE: It's imperitive that you DON'T use the old newnews.pl script from the beta 5 package as it has a serious bug in the auto-purge department! (ask Tamarian... he'll tell you!). Everything except newnews.pl should be used from the old package for beta 5.

As for bandwidth, Chris is right. You can use the the size of the database as a rough guide. I haven't actually looked, but my vserver account only allows 20GB/month and I have 2 sites with 75,000 page views pulling about 80 newsgroups between them and I haven't had a problem. You'll be using megabytes rather than gigabytes.

[Gilby]

I am using multiple servers and posts to one of the servers does not make it to the newsgroup. Looking at the output of running the newnews.pl script, all outgoing posts are trying to be posted on the first newsgroup it processes which for posts that are going to a newsgroup that is not on that server, it means it doesn't make it to the newsgroup but says it was sent. Disabling all the newgroups on one server makes it where the posts go to the right server (but tries to send the messages for the disabled newsgroups too). It does the same for inserting replies where all of them are done on the first newsgroup, but they go to the correct forum, so that's not a big deal.

This has happened since I upgraded to the vb2.0b5 version and I just tried it with the newest newnews.pl script and it occurs with this one too.

I have not investigated further... I would assume it's a coding problem or a problem with the database tables (something not upgraded properly). Does this happen for others that are using multiple servers?

Here is the output from newnews that I get:
Connecting to news.tc.umn.edu... Connected
Sending authentication info... Authenticated and logged in
Getting article batch from rec.sport.unicycling
No new messages in rec.sport.unicycling
inserting new threads into forums
finding replies...
checking for orphans...
Processing outgoing messages
Posting message by Gilby to webdiscuss.test... Sent

You can see that it is trying to post to "webdiscuss.test" while processing for "rec.sport.unicycling". The "webdiscuss.test" newsgroup is disabled and also is on another server. This post did not make it to the newsgroup even though it says "Sent".

[fastforward]

Originally posted by Gilby
I am using multiple servers and posts to one of the servers does not make it to the newsgroup. Looking at the output of running the newnews.pl script, all outgoing posts are trying to be posted on the first newsgroup it processes which for posts that are going to a newsgroup that is not on that server, it means it doesn't make it to the newsgroup but says it was sent.
I've had a quick look and can't see anything obvious, so I'll need to spend some time trying to find out what's wrong. I should have something by tonight.

One point to note though. The sent message only means the server accepted the message. The server will not check to see if it's a valid group first. That validation should really be handled by the script. However, as the groups are pre-defined within vB the chance of sending to an invalid group are practically zero... except in this case :rolleyes:

[fastforward]

I think its due to misplaced parenthesis in the WHERE clause of the post_outgoing() function. As it is at the moment, it will:

either (get messages with no refs) OR (get aanything with a subject that doesn't start with Re: and is in this group)
Therefore it will match any message that has no refs. regardless of whether it's in the right group.

What it should be doing is:

(get messages with no refs or ones that have a subject that doesn't start with Re:) AND (also make sure they are in the newsgroup that is currently being processed)

Here's the fix:

Replace line 750 which currently reads:

my $q2 = db_fetch("SELECT poster,newsgroup,subject,refs,body,threadid,postid FROM usenet_outgoing WHERE (refs <> ' ') OR subject NOT LIKE 'Re: %' AND newsgroup = ". $dbh->quote($$newsgroup->{'newsgroup'}));

With this line:
my $q2 = db_fetch("SELECT poster,newsgroup,subject,refs,body,threadid,postid FROM usenet_outgoing WHERE (refs <> ' ' OR subject NOT LIKE 'Re: %') AND newsgroup = ". $dbh->quote($$newsgroup->{'newsgroup'}));

Let me nkow if that fixes it. I can't actually test it at the moment coz I'm busy at work. :p

[fastforward]

The fix mentioned above is now packaged up and the release number is 2.8.2. There was also another little problem with the logic that handled re-importing of forum generated posts that didn't exist in the forum. Unless you had the option to re-import turned on, you wouldn't have noticed this.

[Gilby]

Originally posted by fastforward
Let me nkow if that fixes it.

Works great now! Thanks.

[Mark Hewitt]

<problem solved post removed>

[Mark Hewitt]

When upgrading the board I need to be able to take backups of the database in case things go wrong.

I tried this on moving to a test database but got this

ERROR 1064 at line 2848: You have an error in your SQL syntax near 'ord(ord), KEY isusenetpost(isusenetpost)
) TYPE=MyISAM' at line 26


This is a serious error because it means I can no longer take backups of my database :(

Can anyone tell me a fix or point me to the instructions as to how to fully uninstall the hack from the database? I've got the empty_usenet.pl script but want to make sure it will run with the latest version of the hack, since I can't backup my database :(

[fastforward]

Originally posted by Mark Hewitt
When upgrading the board I need to be able to take backups of the database in case things go wrong.

I tried this on moving to a test database but got this

ERROR 1064 at line 2848: You have an error in your SQL syntax near 'ord(ord), KEY isusenetpost(isusenetpost)
) TYPE=MyISAM' at line 26


This is a serious error because it means I can no longer take backups of my database :(

Can anyone tell me a fix or point me to the instructions as to how to fully uninstall the hack from the database? I've got the empty_usenet.pl script but want to make sure it will run with the latest version of the hack, since I can't backup my database :(
What exactly are you trying to do? What code generated that error? I can still take backups of my database.

To uninstall the hack, simply upload all your original vB files and drop the columns and table that were added to the vB tables. You can see what these were in the installation scripts.

[fastforward]

This release affect newnews.pl only. It fixes a few minor errors.[list=1] Problem with email notification not using the vB templates correctly. Fixed autopurge option to correctly clean out the searchindex table. Added additional checks to all queries that delete or update data to ensure that only usenet groups are affected.[list]
You can just overwrite your newnews.pl with the attached file. The new file is also in the full package.

The next release will have a lot of changes that eliminate the need for many of the code edits to vB files. I have already got it working without the need to edit newreply and newthread.php which also means move,copy,split etc will work without problems for any thread.

The other major change is the removal of the option to import usenet users. I think this is a waste of time and space and serves no real purpose except the ability to search by username. For the extra code edits and problems that this introduces, I don't think it's worth it. Anybody have a problem with that?

My aim is to limit the code changes to a state that allows vB upgrades to happen with no loss in functionality to either vB or the hack (except maybe some inevitable display issues such as count of usenet posts etc)

[tamarian]

Originally posted by fastforward

The next release will have a lot of changes that eliminate the need for many of the code edits to vB files. I have already got it working without the need to edit newreply and newthread.php which also means move,copy,split etc will work without problems for any thread.

Can't wait for this! :)

[Mark Hewitt]

Originally posted by fastforward
What exactly are you trying to do? What code generated that error? I can still take backups of my database.
[quote]

Just dumping the database and attempting to reupload it to an empty database.

[quote]
To uninstall the hack, simply upload all your original vB files and drop the columns and table that were added to the vB tables. You can see what these were in the installation scripts.

Thanks, I'll do that.

I did notice one bug which you may or may not be aware of, when I tried to 'update counters' for last posters etc, it threw a database error when it found a usenet post with a / or a ' character.

I can't wait until your next version of this hack :cool:

[TechTalk]

If you make a version of this that can run without editing any vbfiles (other than templates) i will bow to you.

I really want to use it but I refuse to edit vb files so when something goes wrong their the only ones to blame :)

Cant wait for the next version :)

[fastforward]

This is a first release of the new newnews.pl lite :)

RC2 came out a bit sooner than expected, so I rushed out an early version.

The good news:

Only one vB code edit to make! And that is just to add a link for the control panel on the admin/index page.
Discovered and fixed some more bugs and efficiency issues
Reduced the number of additional columns added to vB tables


The bad news:

You have to work with your templates to hide unnecessary fields such as post counts and location etc. As there are no conditional statements or constructs alowed in templates you will have to create a new styleset and assign it to the usenet forums. This could actually be seen as an advantage.
Due to the unexpected release of RC2, this release doesn't provide a way to display a seperate count of usenet posts on the front page. Neither does it allow posts to be hidden from the 'new posts' search. I will try to figure a way to do this without editing the code.
Finally, there is no extra permission flag to prevent posts propogating to usenet. This option was a little pointless anyway and could be confusing. If you don't want people posting to usenet then just don't let them post in the usenet forum. Use the vB permissions for it.


More good news: I believe it's possible to have a fully featured usenet gateway with zero code edits to vB files.

Once again, this release was a little rushed. I believe the posting and threading are working correctly, but there may be some other issues I've overlooked so use it with care. It shouldn't really affect any of the vB functionality as we don't touch the code anymore.

One more thing, as mentioned in an earlier post, the 'import user option' is gone.

The package has been updated in the first post of this thread or you can get it here (http://britishexpats.com/download/usenet_gateway.tar.gz).

If you want to continue using the old one with all the code edits, you can still get it by calling up a directory listing at http://britishexpats.com/download and picking the version you want.

[Mark Hewitt]

Since this doesn't edit vbulletin code anymore is the last thread title "bonus" still included (I assume not).

I use this on my board, is there anywhere I can get instructions as to the code changes needed to implement this on it's own?

[fastforward]

Originally posted by Mark Hewitt
Since this doesn't edit vbulletin code anymore is the last thread title "bonus" still included (I assume not).

I use this on my board, is there anywhere I can get instructions as to the code changes needed to implement this on it's own?
Ah.. I didn't think anyone was using this. I'l get the instructions out later today. It's quite easy and only needs editing in one place (indexpost()) if I remember correctly.

[tamarian]

Fastforward, great job. I just looked up the instructions. This is virtually no code change? If I use phpMyAdmin to enter my news server info and options, or if the settings are entered in newnews.pl, no code changes are necessary?

[fastforward]

Originally posted by tamarian
If I use phpMyAdmin to enter my news server info and options, or if the settings are entered in newnews.pl, no code changes are necessary?
Yep. That's right. I did actually have a go at changing the config page to work standalone which would mean you wouldn't even have to make the one little change left, but it was getting late and I broke it :) I'll give it another go if I get time before RC3 or vB final is released.

I've figured an easy way to get the usenet post counts and any other config setting you make in the usenet control panel into a global variable that can be referenced in templates. I can just append them to the options template during each news pull. The only drawback to that is if you update the vB config using in between news pulls, the usenet variables will be overwritten until the next news pull set them again. It won't be a problem as long as you remember to resubmit the usenet config straight after you update the vB config. And if you really can't remember to do that, it would only be one little optional code change to functions.php to automate it. I'll try to find another way, but that method is favourite at the moment.

I still haven't figured out a way to eliminate the usenet posts from 'new post' searches without code edits. I'm sure there must be a way though :)

[Mark Hewitt]

I've tried the new version but all the posts have the date/time of when they were inserted into the database rather than the correct date when they were posted.:(

[fastforward]

Originally posted by Mark Hewitt
I've tried the new version but all the posts have the date/time of when they were inserted into the database rather than the correct date when they were posted.:(
That is how it's supposed to work. Otherwise, they won't be threaded correctly when local posts are in the same thread as usenet posts.

If you want it the old way then you will have to go back to the old method of editing the vB code in newreply.php, newthread.php, postings.php and functions.php. Which would you prefer?

The time of posting is not very reliable anyway. It's not even the NNTP time. It's based on the users PC from which the post was made. With this method, the post times are actually going to be more consistent. Plus. if you're pulling news 2 or three times an hour, there won't be much discrepency between the original post (except when the original post was wrong).

[Gilby]

Originally posted by fastforward
That is how it's supposed to work. Otherwise, they won't be threaded correctly when local posts are in the same thread as usenet posts.

I like the old way better, where the date of the post is the same as the date in the message on the newsgroup. Maybe some date checking would be appropriate such as making sure the date isn't in the future and that it's not older than a few hours or since the last time newnews was run, whichever is older.

Let's say that either the news server or your own server was down for a while. You don't want to have the messages that were not loaded earlier being the same date as all the others. Sometimes it takes a while for messages to make it to the a particular news server and they might come out of order too.

[fastforward]

With this method, posts will appear just as they do using a newsclient against a newserver. The news server simply displays them as it receives them.

If the date of the news post is used and placed into the dateline field, that is how vB orders the posts. This means they will NOT be shown in the correct order due to the time taken for news to propogate across usenet. You will get usenet posts coming in that are placed in the middle of threads. Also, they may not show up as new posts as they may be dated before your last visit. It becomes impossible to keep track of long threads.

The other problem is when a local reply is posted in a usenet thread. You will also get a whole bunch of usenet posts placed above it when the next run occurs. The local poster gets an email telling them someone has replied and can't see any posts after their one. Very confusing and frustrating.

These example aren't isolated occurances. It happens all the time. In fact, the very first version of this hack did exactly what you are asking for. It was changed for the reasons above.

The way it was handled in the more recent versions was by using the 'seq' (sequence) field. The thread was orderd using this column which meant everything was shown in the right sequence (although the message dates were not necessarily in order). This was achieved by editing the showthread, newreply, newthread and posting.php.

As far as checking goes, this was also done in the previous version to minimize the message dates appearing strange and out of sequence, and also to make sure posts were shown as new in a timely fashion. But, it's only of use when combined with the vB code changes.

Basically, you have a 3 choices.

Use the news post date with no vB code changes and have posts that you may never see as they will be inserted in the middle of threads and not be marked as new.
Use the method of setting the date to the date they are inserted into the forums. (Just as a real posts do). The posts are in the correct order, but have a 'wrong' date.
Use the newspost date along with all the code changes that are necessary to support it and the inability to upgrade to newer vB versions without breaking the hack, vB, or both.

[chrispadfield]

trust me, you want to stick with this way! getting posts going above native VB posts is incredibly confusing and as fastfoward says it happens all the time. Newsgroups themselves don't sort by date/time, VB shouldn't do either for newsgroup posts and threads.

[Mark Hewitt]

Use the news post date with no vB code changes and have posts that you may never see as they will be inserted in the middle of threads and not be marked as new.

That is actually how I would like it to work, would it be possible to set this as an option in the usenet.php control panel as I really don't like the way it is done now.


OK, that "bug" now found to be a "feature" :). I have another one for you :rolleyes: !

The names don't appear in the individual posts. Even though they do appear in the thread starter and last poster columns on forumdisplay.

Also I don't know if this is to do with your code but threading on outgoing posts doesn't seem to be working correctly. There was a long thread with 5 replies and I clicked reply and replied to the message. I expected the post to be in reply to post number 5 instead it appeared in reply to the first post.

[fastforward]

Originally posted by Mark Hewitt
Also I don't know if this is to do with your code but threading on outgoing posts doesn't seem to be working correctly. There was a long thread with 5 replies and I clicked reply and replied to the message. I expected the post to be in reply to post number 5 instead it appeared in reply to the first post.
This isn't possible without some major code changes. vB is not a threaded bulletin board. If vB doesn't know or care which post is being replied to, it's impossible for an external script to figure it out.

It would be possible to figure out if an individual post was being replied to by using the quote feature, but that would mean more vB code changes and it would only work if the quote was used.

This hack has always only 'threaded' incoming messages by placing them in the correct sequence and a logical order. Outgoing messages have always been in reply to the first post.

As for the name not appearing, I'll have to look into that some more. They seem to be showing up on mine.

[Mark Hewitt]

This isn't possible without some major code changes. vB is not a threaded bulletin board. If vB doesn't know or care which post is being replied to, it's impossible for an external script to figure it out.


Ah right, that's a pity because it will look quite strange on usenet. (There quite a fickle lot over on the newsgroups). Can't newreply.php be modified to include a "in reply to" tag which would handle this. (code changes aren't that bad really!).


As for the name not appearing, I'll have to look into that some more. They seem to be showing up on mine.

Thanks, see example here -
http://www.motorsportforum.com/vb22/upload/showthread.php?s=&threadid=235

[fastforward]

Originally posted by Mark Hewitt
Can't newreply.php be modified to include a "in reply to" tag which would handle this. (code changes aren't that bad really!).

It could. But the whole point of this 'new' usenet hack is to eliminate all code changes to ease support issues for everyone and allow timely upgrades to new vB versions. It also means I don't have to stay up until 4am the day Jelsoft releases a new version :)

Remember this new version is still in it's infancy after stripping out all the code changes. A lot of the missing features will be readded... and still with no code changes. I just need time to figure out how.

By the way, I can add an option for the usenet post date issue. But remember; new posts may be placed before recent posts. The dates will obviously be in sequence, but you may receive the newer post first. The real problem is when local posts are entered into a thread along with usenet posts. You will nearly always get usenet posts placed before it. I'll add the option with a disclaimer.

[tamarian]

Fastforward,

Here's a suggestion. No vb code changes, and minimal usenet customization can be the basic usenet hack. A list of mods to the hack can be updated, by you and other experienced users of the hack. Like a text in the hack zip with a format like:

1. Search: To exclude usenet posts from being displayed in view new, find this in search.php, replace with, etc.

2. Stats to eclude from stats, do this...

bla bla. Each needed enhancement can be hacked by itself, while the simplest usenet hack won't need any code changes.

I can see this hack taking a lide of it's own!

[Mark Hewitt]

I would be in favour of that but I doubt fastforward would since he is doing this for free after all!

However another thing I found was that wereas previously the usenet posts didn't appear in my overall post total they do now, this is bad.

[tamarian]

Originally posted by Mark Hewitt
I would be in favour of that but I doubt fastforward would since he is doing this for free after all!


We probably won't go through another repetitive upgrade cycle like vb1.6-vb2.0 for a while. Some mods to the Usenet hack are quite simple, especially if you hard code them instead of using the CP. Like adding "AND forumid <>" in search to control search.

I actually use the 2.8.2 of the hack on top of RC2, so it still behaves like it used to, and will install 2.9 when vb2 is released, with mods for stats msg count and seach behaviour on the usenet messages.

[fastforward]

Tamarians suggestion isn't too bad actually. It will still only be one version. It will just be up to the user how many of the optional code changes they wany to make to integrate it with vB. It will make the newnews.pl a bit more complicated due to the additional if/elses etc but it shouldn't be a big deal. I still intend to drop the 'import user' function.

Does that sound like a resonable solution?.

[tamarian]

Originally posted by fastforward
Tamarians suggestion isn't too bad actually. It will still only be one version. It will just be up to the user how many of the optional code changes they wany to make to integrate it with vB. It will make the newnews.pl a bit more complicated due to the additional if/elses etc but it shouldn't be a big deal. I still intend to drop the 'import user' function.

Does that sound like a resonable solution?.

VERY much so!

Thanks :)

[Mark Hewitt]

More than I ever hoped for :D

[Mark Hewitt]

Theres only I few things that I would need in the next version before I can implement it on my board.

Fix the blank name on posts bug
Option to show date as when posted, rather than imported
A way to exclude usenet posts from the total post count. (and from pretty much everything else as well, but the post count is most important).

Cheers :cool:

[fastforward]

Originally posted by Mark Hewitt
Theres only I few things that I would need in the next version before I can implement it on my board.

Fix the blank name on posts bug
Option to show date as when posted, rather than imported
A way to exclude usenet posts from the total post count. (and from pretty much everything else as well, but the post count is most important).

Should be all done by tonight.

[fastforward]

This is the first release of the new style usenet gateway that works without any code edits but allows you to optionally integrate it more tightly into vB by making code edits. You can make as many or as few code edits as you like. The script will still work.

This release details enough edits to bring back the functionality that was stripped out in the last release. More integration/options will be added as and when people request them. Two of my next features will be the ability to download from the binary groups and provide a link to the file from the description usually found in the '01' section of the usenet article and an option to recieve email from usenet via the PM system.

Here's what this one can do:

Without code changes:

full control panel integration
email notification to usenet replies
emoticon translation into vb icons
vb code removal or conversion prior to posting to usenet
hyperlinked urls in messages & color coded and italicized posts
vB style quote to usenet style quote conversions for outgoing posts
logging of outgoing posts
support for multiple news servers
support for seperate footers per forum in outgoing posts
multi-language (selectable) handling of quoted MIME printable headers (for all those funny foreign characters)
flexible spam control and replacement variable options for both incoming and outgoing messages
configurable auto-purge option
global option for users to show their email address if user has selected that his/her email be displayed normally
global option to enable/disable user signatures on outgoing posts.
referencing usenet post count in templates (total posts will also include usenet posts)
optional time source for usenet posts (time entered into forum vs time user posted to usenet)
option to turn off indexing for usenet posts


With replacement variables and/or template changes:

user selectable email address in their outgoing usenet posts (may be different than normal one in profile)
suppression of unnecessary info in usenet posts, ie location, post count etc.
configurable indented quotes in incoming usenet posts


With code changes:

suppression of usenet post count from totals in templates
auto refresh of global usenet variables to ensure post counts etc are always up to date
optional suppression of usenet post count from totals in stats
better ordering and sequencing of usenet posts when using the NNTP posting date as time source
the 'last active' thread hack a la UBB.


Download the latest version (http://britishexpats.com/download/usenet_gateway.tar.gz)

Once again it's late and I'm tired, so treat this release with care. Let me know of any bugs asap and I can release a true 'release version' to coincide with the vB final. You'll probably notice I've changed the version number to date. With the frequency of updates this makes more sense and is easier to track... maybe :)

I also intend to move this hack and all discussion over to vbdev.org at the next release.

The latest version of this hack will always be in this first post of the thread.

[tamarian]

As always, you rock, fastforward :)

vbdev.org? Did you mean:

http://www.vbulletin.org

Search, and view new posts disabling? I personally hard code it, since I'm only interested in one usnet group.

I love it, and will install this weekend and report bugs (if any) :)

[fastforward]

Originally posted by tamarian
vbdev.org? Did you mean:

http://www.vbulletin.org

yeah... whatever :) I must have got confused with ubbdev :eek:

Search, and view new posts disabling? I personally hard code it, since I'm only interested in one usnet group.

ooops. I forgot about that one. I'll add that tomorrow.

[Mark Hewitt]

I can run newnews.pl and it appears to run through correctly. However after it does I get these errors on every page of my forums


Parse error: parse error in ./global.php(55) : eval()'d code on line 63

Warning: Cannot add header information - headers already sent by (output started at ./global.php(55) : eval()'d code:63) in ./functions.php on line 1172

Warning: Cannot add header information - headers already sent by (output started at ./global.php(55) : eval()'d code:63) in ./global.php on line 131


When I load my forums home page it gives the above errors and only displays the categories and no forums. It also gives a database error when trying to run showthread.


Database error in vBulletin: Invalid SQL:
SELECT IF(votenum>=,votenum,0) AS votenum,
IF(votenum>= AND votenum > 0,votetotal/votenum,0) AS voteavg, icon.title as icontitle,icon.iconpath,
thread.threadid,thread.title,lastpost, forumid,pollid,open,replycount,postusername,postus erid,
lastposter,thread.dateline,views,thread.iconid,not es,thread.visible,sticky,votetotal,attach
FROM thread
LEFT JOIN icon ON (icon.iconid = thread.iconid)

WHERE thread.threadid IN (0)
ORDER BY sticky DESC, lastpost DESC

mysql error: You have an error in your SQL syntax near 'votenum,0) AS votenum,
IF(votenum>= AND votenum > 0,votetotal/v' at line 2
mysql error number: 1064
Date: Wednesday 02nd of May 2001 06:35:06 AM
Script: /forums/forumdisplay.php?s=&forumid=3
Referer: http://www.motorsportforum.com/forums/forumdisplay.php?s=&forumid=33


However if I go to /admin/options.php and resubmit the options everything goes back to normal :confused:

[fastforward]

What code changes did you make? Did you go through and re-check the tables to make sure all columns and indexes are as specified in the new versions. Alao, did you make sure you removed any existing code changes you may have made?

[Mark Hewitt]

Originally posted by fastforward
What code changes did you make?


I implemented all the code changes you specified.


Did you go through and re-check the tables to make sure all columns and indexes are as specified in the new versions.


Yes, I deleted all my current usenet forums and removed the usenet_ tables. I then made sure the extra feilds matched up. Think it was only regpost I needed to add. I then created and populated the new usenet tables.


Also, did you make sure you removed any existing code changes you may have made?

I implemented it from a clean copy of RC2 which I was previously running the last version of the hack on.

Hate to heap more problems on you but the NNTP posting time option doesn't seem to work, still uses local time.

Apart from that it works a treat :D

[fastforward]

I'm working on it. I'll get back to you.

:rolleyes: If only there were no users. life would be so much easier for developers :D

[Mark Hewitt]

If only there were no users. life would be so much easier for developers :D

I hear ya. :cool: Especially ones that don't know how to run a Java class (long story)

[fastforward]

OK...

The parse errors may have been because it was the first time the script was run. The way it works now is it appends any usenet variables that need to be referenced by vB to the 'options' template. This means they can be accessible through layout templates without code changes. The variables causing problems are the post counts which were not quoted. If these are NULL, then it will generate a parse error as the resulting code would like this: ' $usenettotalthreads= ' instead of ' $usenettotalthreads="" '. That's easy to fix.

I can not reproduce your time problem. When I set NNTP in the options all my times show up correctly as the original posting date. Can you reconfirm that this is happening by comparing it against your newsreader? Or have you already done that?

[Mark Hewitt]

The parse errors may have been because it was the first time the script was run. The way it works now is it appends any usenet variables that need to be referenced by vB to the 'options' template.

This sounds about right, especially as everything was restored when I resubmitted the options. However I did run the script twice and the same problem happend both times.


I can not reproduce your time problem. When I set NNTP in the options all my times show up correctly as the original posting date. Can you reconfirm that this is happening by comparing it against your newsreader? Or have you already done that? [/B]

Looking back on it, it's possible the option wasn't set on the second time I ran it. As half the posts seem to be okay. So ignore that part :)

[fastforward]

A few bugfixes and the addition of the 'exclude usenet posts from new post search' option.

Take note of item number 4. If you have already installed the last version, you MUST change this column type to SIGNED for correct handling of outgoing posts.

Added option to exclude usenet posts from 'Get New Posts' search.
Fixed bug causing parse errors during first run.
Fixed bug causing newnews.pl errors 'e.field invalid column' error if custom profile field for outgoing email isn't added.
Fixed incorrect column type for isusenetpost in post and thread tables. They should be TINYINT(2) SIGNED.

This should be one step closer to a bugfree version. Time is running out, vB 2.0 will be out any day :)

I'll delay the move to vbulletin.org until the final release.

Download latest version (http://britishexpats.com/download/usenet_gateway.tar.gz)

[Mark Hewitt]

Originally posted by fastforward
[B] Fixed bug causing parse errors during first run.

:( I'm still getting the same errors after a run that I detailed above :(

[Mark Hewitt]

I've looked at the options template and after it has just run it has

$totalusenetposts=5902;
$totalusenetthreads=18;

After I go back and resumit the vB options page is changes to

$totalusenetposts="5902";
$totalusenetthreads="18";

i.e. it now has quotes

For temporary fix I've commented out generateoptions(); as I don't use total threads/posts anyway, and now all seems well.

(another edit). The NNTP post time seems to work for the first few posts then gives up and goes back to local time.

[fastforward]

I think caching causes problems for this template update function aswell. I think the best solution will be if I remove it for the threadcounts and you will have to make a code edit to use them. It's only a one liner anyway.

I still can't reproduce your posting date problem. It's actually only 1 little line in the script that decides the date. When the script starts up, it reads all the config from the tables and then when it gets to the load forums bit it checks to see whether it should replace the nntp date with the local date. So the default is actually NNTP date. I'll keep trying to reproduce it though.

[fastforward]

I've just looked at the code and the quotes are gone again! I think this because I started making the the edits on my test board while at work and then not transfer them to the distribution copy when I got home. All the other fixes I mentioned were still there, it's just the quuote issue around the post counts (which I'm removing anyway).

[fastforward]

OK... sorry about all these posts. I have just found the problem with the date thing. The problem is obvious and I have no idea how I missed it! The conversion is only done when a thread starter is loaded. For replies I still had a hard code of '$dtm=time();' in there... ooops

Sorry about that.

[Mark Hewitt]

Ah right :cool: good to hear it's not my misconfiguration again :).

[fastforward]

Here's another one that attempts to fix the recent issues.

Eliminated parse errors by removing the function that appended to the vB templates. It was too unreliable and prone to error. In order to display the usenet counts on template pages you must now make a code edit. There is also a requirement to make one common code change if any of the optional code edits are made.
Fixed the NNTP/LOCAL time issues. (really this time... honest!)
There is also an addition to the code change instructions if you are using the lastactivethread hack.


Download v20010504 for RC2 (http://britishexpats.com/download/usenet_gateway.tar.gz)

[fastforward]

I messed up again... I forgot to include the new usenet.php file in the package! I've edited the previous post and changed the version and download link. All should be OK now.

[Mark Hewitt]

When I run newnews.pl I get this


Fetching article body 64980... OK
Processing article batch...
Use of uninitialized value in numeric gt (>) at newnews.pl line 216.
Use of uninitialized value in concatenation (.) at newnews.pl line 228.
Use of uninitialized value in concatenation (.) at newnews.pl line 228.
Use of uninitialized value in concatenation (.) at newnews.pl line 228.
Use of uninitialized value in concatenation (.) at newnews.pl line 228.
Use of uninitialized value in concatenation (.) at newnews.pl line 228.
Use of uninitialized value in concatenation (.) at newnews.pl line 228.
Use of uninitialized value in concatenation (.) at newnews.pl line 228.

Query failed:
INSERT LOW_PRIORITY IGNORE INTO usenet_ref (msgid,ref,cnt,dtm) VALUES ('<9abt4l$eqr$4@dyfi.aber.ac.uk>','<99pjhv$6tt$1@dyfi.aber.ac.uk>',6+1,),('<9abt4l$eqr$4@dyfi.aber.ac.uk>','<%k2w6.3328$W51.2310054@e3500-chi1.usenetserver.com>',6+1,),('<9abt4l$eqr$4@dyfi.aber.ac.uk>','<3AC0F88A.312BF9B7@crosslink.net>',6+1,),('<9abt4l$eqr$4@dyfi.aber.ac.uk>','<3AC3FEDC.4043@i-plus.net>',6+1,),('<9abt4l$eqr$4@dyfi.aber.ac.uk>','<99vgrr$g0k$2@dyfi.aber.ac.uk>',6+1,),('<9abt4l$eqr$4@dyfi.aber.ac.uk>','<290320010806569457%scott@nospam.winders.com>',6+1,),('<9abt4l$eqr$4@dyfi.aber.ac.uk>','<9a9ehk$5fu$5@dyfi.ab',6+1,)

DBD::mysql::db do failed: You have an error in your SQL syntax near '),('<9abt4l$eqr$4@dyfi.aber.ac.uk>','<%k2w6.3328$W51.2310054@e3500-chi1.usenetse' at line 1 at newnews.pl line 576.


I try running it again until it gets to that particular post and it fails again :( Previous to that though I had imported some posts and was working correctly.

[fastforward]

What's the size and type of your ref column in the usenet_ref table. Try making it bigger.

[Mark Hewitt]

It was varchar(128).

I've tried changing it to varchar(255) but get the same error.

[Mark Hewitt]

If you need to test it the import was

Getting article batch from rec.motorcycles.racing
Fetching headers of articles 64663 to 64980...

[fastforward]

It' the usenet date/time. It either doesn't exist (unlikely) or it's malformed in the usenet post header. This is one of the reasons I went to local time. Have a look at the article on the server and see what the date looks like. I can't see it as your article numbers mean nothing to my server. As a workround you can add the following code at line immediately above line 216 that reads: if ($dtm > time()+600) { $dtm = time(); }

if (!$dtm) { $dtm = time(); }

That is an addition, NOT a replacement.

[fastforward]

Note the little change I just made to that change. It should be !$dtm.

[annams]

First of all, this is a wonderful hack!!

Now, on my web server I did not have Email::Find and MIME::WordDecoder and I installed both the modules. After that I am encountering the following error:

defined(@array) is deprecated at /usr/home/dsw/local/lib/perl5/site_perl/5.6.0/Net/DNS.pm line 137.
(Maybe you should just omit the defined()?)

Please help!!!

[fastforward]

Originally posted by annams
Now, on my web server I did not have Email::Find and MIME::WordDecoder and I installed both the modules. After that I am encountering the following error:

defined(@array) is deprecated at /usr/home/dsw/local/lib/perl5/site_perl/5.6.0/Net/DNS.pm line 137.
(Maybe you should just omit the defined()?)

That error is actually being generated from the Net::DNS module. That module is not part of the hack. I believe the DNS module is patr of the libnet bundle. Try upgrading the libnet bundle using CPAN.

If that fails, you may want to try your suggestion. Find the line in DNS.pm that it talks about and change it there. Try just declaring it as @array; and see what happens.

In the meantime, I'll see if I can find any reference to this problem anywhere else.

[annams]

That was just a warning! Since no processing was happening after that I thought it was an error. But then I found that I had both groups disabled. Now, I got everything working fine.

Thanks a bunch for the hack!!

Btw, I had to take a paid registration with usenet.com to get the newsfeed. Though I like the fact that it has a spamfree news server. Is there a free or cheaper newsfeed?

[fastforward]

Originally posted by annams
That was just a warning! Since no processing was happening after that I thought it was an error. But then I found that I had both groups disabled. Now, I got everything working fine.

If you remove the -w switch from the shebang line in newnews.pl that warning will go away. At least you won't have to look at it each time :)

Btw, I had to take a paid registration with usenet.com to get the newsfeed. Though I like the fact that it has a spamfree news server. Is there a free or cheaper newsfeed?
There are free ones around, but you have to look hard. They are usually unreliable, slow and carry a limited set of groups. You might be able to find decent $5.95 one for just the text groups but $10 seems the norm for a decent account.

[Mark Hewitt]

I've made all the changed you specified and in addition I've switched the option to use local server time. However the same error still results :confused:

[Mark Hewitt]

I removed the newgroup that was causing the trouble but it just happened on another group.


Fetching article body 27334... OK
Processing article batch...
Use of uninitialized value in numeric gt (>) at newnews.pl line 216.
Use of uninitialized value in concatenation (.) at newnews.pl line 228.
Use of uninitialized value in concatenation (.) at newnews.pl line 228.
Use of uninitialized value in concatenation (.) at newnews.pl line 228.

Query failed:
INSERT LOW_PRIORITY IGNORE INTO usenet_ref (msgid,ref,cnt,dtm) VALUES ('<philip.
remove.orr.629.011B100A@bigfoot.com.spiced.ham.in. a.can.au>','<3ab213a3@news.ipr
imus.com.au>',2+1,),('<philip.remove.orr.629.011B100A@bigfoot.com.spiced. ham.in.
a.can.au>','<3ab21537@occy.pnc.com.au>',2+1,),('<philip.remove.orr.629.011B100A@
bigfoot.com.spiced.ham.in.a.can.au>','<3ab2c1ce@grissom>',2+1,)

DBD::mysql::db do failed: You have an error in your SQL syntax near '),('<philip
.remove.orr.629.011B100A@bigfoot.com.spiced.ham.in .a.can.au>','<3ab21' at line 1
at newnews.pl line 576.
bash$

[fastforward]

There's no need to set it to local time. The change just checked whether the $dtm field was populated with the nntp time and if it wasn't use the local time.

There is no doubt the error is being caused by the $dtm field not being populated. The reason this happens is because the time in the header is missing or malformed. The warning: 'Use of uninitialized value in numeric gt (> ) at newnews.pl line 216' is due to the $dtm field being null. (an advantage of 'use strict' and the -w flag) confirms this. Also you can see the dtm field is missing in the sql statement.

What news server are you using and what groups are giving the most problems. I'll try them and add some more error checking.

To confirm the problem, do this: Find line 299 that currently reads:

Add the following line directly ABOVE it
[code]
if (!$d) { $d=time(); }

So the resulting block of code around there will be:

if (/^Lines: (.*)/) { $l = $1; }
}
if (!$d) { $d=time(); }
($p,$e) = parse_from($f);
push (@a,{ from => $f,

Another quick fix is just remove the NOT NULL constraint from the usenet_ref table. This may have other unwanted side effects though :)

[maxc]

Hi....fastforward, Thanks for your hack...
But I have some questions, When I used spam control,it always appear a message as below:

Warning: Cannot add header information - headers already sent by (output started at /home/httpd/html/forum/admin/adminfunctions.php:18) in /home/httpd/html/forum/admin/usenet.php on line 231

how to slove this problems.please...

and in Options -> Character set for MIME decoder, it appear

Parse error: parse error, expecting `']'' in /home/httpd/html/forum/admin/usenet.php(55) : eval()'d code on line 2

Thanks for your help.... :) :) :)

[maxc]

Originally posted by fastforward

To confirm the problem, do this: Find line 299 that currently reads:

($p,$e) = parse_from($f);

Add the following line directly ABOVE it

if (!$d) { $d=time(); }

So the resulting block of code around there will be:

if (/^Lines: (.*)/) { $l = $1; }
}
if (!$d) { $d=time(); }
($p,$e) = parse_from($f);
push (@a,{ from => $f,

Another quick fix is just remove the NOT NULL constraint from the usenet_ref table. This may have other unwanted side effects though :)

I have already patch my newnews.pl with this method, But it still show the error message below:

Query failed:
INSERT LOW_PRIORITY IGNORE INTO usenet_article (newsgroup,forum,msgid,dtm,subject,poste
r,email,refs,body,msgnum,nntpposter,ord,threadid,p ostid) VALUES ('tw.bbs.comp.hardware'
,7,'<3f0Mhc$Zy5@bbs.cis.nctu.edu.tw>',982109749,'Re: ?бЦU?찪???㦳??window2000??????
?]?w???D????????','?A???N?Ъ??p?\\','wuuchung.bbs@bb s.cis.nctu.edu.tw',NULL,'==> ?b ens
.bbs@zoo.ee.ntu.edu.tw (?S?n?Z???F) ???峹??????:\n> ?N?O?ڭ̲oADSL\n> ?M???D???A??W
IN98?t??\n> ?ڬO??WIN2000?t??\n> ?Q??sygate?ӤW??\n> ???O?ڲ{?b?u???WB
BS\n> ?Ӥ????s?WWWW\n> ?аݭn?p???]?w?~???ѨM?H?H?H\n>
??ӷP?E????????\n\n ???ӬODNS?S???]?w?n.......\n--\n\n ?ڬO?ĤT?????N?ЧШk,?{?b?b?x?
_???s?????A???????N??,?????????????N??.\n\n ?Ѳ?.???f.?v??.?q???w??.?????Z?OCS?????D,?
ڳ?w???A?̨ӻP?ڤ@?_?Q??........\n--\n * Origin: ?? ???q?j?Ǹ??T???Ǩt BBS ?? <bbs.cis
.nctu.edu.tw: 140.113.23.3',245055,'wuuchung.bbs@bbs.cis.nctu.ed u.tw (?A???N?Ъ??p?\\)'
,-1+1,0,0)

DBD::mysql::db do failed: You have an error in your SQL syntax near 'wuuchung.bbs@bbs.c
is.nctu.edu.tw',NULL,'==> ?b ens.bbs@zoo.ee.ntu.edu.tw (?S?n?Z' at line 1 at ./newnews.
pl line 579.

bash$

[fastforward]

Originally posted by maxc
Hi....fastforward, Thanks for your hack...
But I have some questions, When I used spam control,it always appear a message as below:

Warning: Cannot add header information - headers already sent by (output started at /home/httpd/html/forum/admin/adminfunctions.php:18) in /home/httpd/html/forum/admin/usenet.php on line 231

That warning will always be there until I get around to rewriting the control panel page. It's on the bottom of my to do list. It doesn't affect the functionality in any way.

and in Options -> Character set for MIME decoder, it appear

Parse error: parse error, expecting `']'' in /home/httpd/html/forum/admin/usenet.php(55) : eval()'d code on line 2

I have no idea. I can't reproduce this. Did you change the character set from the default? I'll look into it.

[fastforward]

For those of you getting the error when inserting into the usenet_ref table; check your my.cnf setting for MAX_ALLOWED_PACKET.

I'll lookat rewriting that section to loop and insert each ref individually. A little less efficient, but it should provide better error trapping and correction.

It must be a configuration issue as I can't reproduce it and I collect over 60 groups on dbforums.com

[fastforward]

maxc:

I just cut and pasted your SQL statement that was giving you and error and it loaded into my database without problems.

Post the structure of your usenet_article table and give me the mysql version.

[fastforward]

Originally posted by Mark Hewitt
I removed the newgroup that was causing the trouble but it just happened on another group.
Query failed:

Looking at the SQL statement, you can see it still isn't picking up the date. Give me the newsgroup name and the date of the dodgy article and I'll have look at it. This error is in no way related to maxc's error by the way.

[maxc]

Excuse me,I am a Taiwanese use Tranditional Chinese,
thus,I surmise I need to edit defalt language character,can
I add Big 5 8bit(Chinese language code) into program,will
it work If I added.it.......

[fastforward]

Originally posted by maxc
Excuse me,I am a Taiwanese use Tranditional Chinese,
thus,I surmise I need to edit defalt language character,can
I add Big 5 8bit(Chinese language code) into program,will
it work If I added.it.......
Unfortunately not. The module I use only supports the ISO 8859 single-byte coded (8bit) character sets for the following:

Latin1 (West European)
Latin2 (East European)
Latin3 (South European)
Latin4 (North European)
Cyrillic
Arabic
Greek
Hebrew
Latin5 (Turkish)
Latin6 (Nordic)

Simply adding to this list will not achieve anything. I believe I need unicode support for Chinese characters (or a module that supports more of the ISO 8859 sets). I'm not sure whether more sets exist yet although I know they are planned.

Sorry about that. I will look around for another module that supports it.

[maxc]

Originally posted by fastforward
maxc:

I just cut and pasted your SQL statement that was giving you and error and it loaded into my database without problems.

Post the structure of your usenet_article table and give me the mysql version.

Ok... I am using MySQL v3.23.37, and My structure of usenet_article table is in the attach file.

Thanks for your help....

[fastforward]

One more thought about the Chinese cahracters:
Try just disabling the worddecoder routine. Do this:

Change line 190 that currently reads

$$art{body}=$wd->decode(join("",@$b));

to this

$$art{body}=join("",@$b);

[maxc]

Originally posted by fastforward

Simply adding to this list will not achieve anything. I believe I need unicode support for Chinese characters (or a module that supports more of the ISO 8859 sets). I'm not sure whether more sets exist yet although I know they are planned.

Sorry about that. I will look around for another module that supports it.

Just Ok! It didn't effect of my forum, Thank you very much! ;)

[fastforward]

Minor release to change the way the custom email address works. This way should be more logical and prevent any user inadvertently setting their options in such a way that will display their email address in outgoing usenet posts.

It also adds some additional error checking in places.

There have been a lot of major changes in the last few versions. This includes changes to table structures and definitions. If you have remnants of installations before the version numbers were changed to a date, you should carefully go through each table and index and check the settings against the instructions. Where possible, you should drop and re-create the table.

This version has been tested on over 200 newsgroups pulling articles from 1 week ago. No errors were seen and not once was there an instance of the fields in article headers not being found. The servers used were onlynews.com and a local DNEWS server.

Download v20010505 for RC2 (http://britishexpats.com/download/usenet_gateway.tar.gz)

[Mark Hewitt]

I've changed to the new version and I'm not getting the errors I was getting before.

However some newsgroups don't seem to be fetching the articles properly.

There are some where I know for a fact there are many many new posts and yet newnews.pl is saying No new messages, together with other assorted errors.


Use of uninitialized value in addition (+) at newnews.pl line 95.
Use of uninitialized value in numeric gt (>) at newnews.pl line 95.
Getting article batch from rec.autos.sport.f1.moderated
Use of uninitialized value in numeric gt (>) at newnews.pl line 99.
Use of uninitialized value in numeric gt (>) at newnews.pl line 99.
No new messages in rec.autos.sport.f1.moderated
inserting new threads from rec.autos.sport.f1.moderated
checking for orphans...
Processing outgoing posts
Processing outgoing posts

[fastforward]

Originally posted by Mark Hewitt
I've changed to the new version and I'm not getting the errors I was getting before.

Use of uninitialized value in addition (+) at newnews.pl line 95.
Use of uninitialized value in numeric gt (>) at newnews.pl line 95.
Getting article batch from rec.autos.sport.f1.moderated
Use of uninitialized value in numeric gt (>) at newnews.pl line 99.
Use of uninitialized value in numeric gt (>) at newnews.pl line 99.
No new messages in rec.autos.sport.f1.moderated
inserting new threads from rec.autos.sport.f1.moderated
checking for orphans...
Processing outgoing posts
Processing outgoing posts

You have something very wrong with your installation. Those errors are telling you it can't find the last message number from the config. Those Particular lines have not been changed since the early versions. The same goes for the date issues you were getting. The reason you don't get date errors now is because it is forced to local time if it's still NULL after trying to get the NNTP date.

I can't reproduce the errors and I've pulled countless newsgroups including rec.motorcycles.racing that was giving you problems.

Have you changed Perl versions recently? Did you do a *complete* fresh install? Is anyone else having these problems.

The only way to troubleshoot your problem would be if I had access to you machine and newserver.

Let me try anaother fresh install on another server and I'll see if I can re-produce the errors. Give me a few hours.

[fastforward]

OK.. Here's another one. I rechecked the entire code and errorchecking. I installed it on a fresh vB on two different servers

FreeBSD with an old version of MySQL (3.22.32) and Perl 5.005_03
Linux 6.1 with the latest MySQL (3.23.37) and Perl 5.005_03


Both installations had the latest required modules from CPAN.
Two newservers were used, onlynews.com and DNEWS. DNEWS is used by a lot of the major news providers including Supernews.

There were no errors found with either configuration on either server.

There realy isn't much else I can do. All I suggest is you make a completely fresh install, DO NOT make any code changes except the bit to add the control panel. Add the basic configuration settings, but leave most of the settings as the default. Then run the script. Change the settings and/or and add the code changes one by one if you need them and see where the errors start occuring.

Download v20010506 for RC2 (http://britishexpats.com/download/usenet_gateway.tar.gz)

[SNA]

Just reinstalled it... it loops after reading in the headers...

Connecting to #######... Connected
Processing outgoing posts
Getting article batch from ######
Fetching headers of articles 1 to 23...
Processing headers for article 2...
Processing headers for article 3...
Processing headers for article 4...
Processing headers for article 5...
Processing headers for article 6...
Processing headers for article 7...
Processing headers for article 8...
Processing headers for article 9...
Processing headers for article 10...
Processing headers for article 11...
Processing headers for article 12...
Processing headers for article 13...
Processing headers for article 14...
Processing headers for article 15...
Processing headers for article 16...
Processing headers for article 17...
Processing headers for article 18...
Processing headers for article 19...
Processing headers for article 20...
Processing headers for article 21...
Processing headers for article 22...
Processing headers for article 23...
Fetching article body 2... OK
Fetching article body 3... OK
Fetching article body 4... Not fetched - Spam
Fetching article body 5... OK
Fetching article body 6... OK
Fetching article body 7... OK
Fetching article body 8... OK
Fetching article body 9... OK
Fetching article body 10... OK
Fetching article body 11... OK
Fetching article body 12... OK
Fetching article body 13... OK
Fetching article body 14... OK
Fetching article body 15... OK
Fetching article body 16... OK
Fetching article body 17... OK
Fetching article body 18... OK
Fetching article body 19... OK
Fetching article body 20... OK
Fetching article body 21... OK
Fetching article body 22... Not fetched - Spam
Fetching article body 23... Not fetched - Binary post
Processing article batch...
Requested 22 messages... 3 not available or rejected.

inserting new threads from cast.testing
checking for orphans...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...
loading 1 replies...

And just keeps going....

I suggest NOT looking for the Lines: ## in the NG posts... Did you know that the posts added FROM the forum dont have a Lines: ## in the header?

Can we disable the option that looks to see if Lines: exsists...

[SNA]

Its fixed now.. I dumped the ref table... But now.. why doesnt the posts from the FORUMS insert a "Lines: ##" line into the header?

[fastforward]

Originally posted by SNA
I suggest NOT looking for the Lines: ## in the NG posts... Did you know that the posts added FROM the forum dont have a Lines: ## in the header?

Can we disable the option that looks to see if Lines: exsists...
You can disable it if you really want to. But you may end up with huge useless posts and possibly binaries.

All posts have the 'Lines:' in the header regardless of their source. It is added by the news server along with all the other stuff that a client news reader can't be trusted to add.

As for the looping problem, have you made several runs and reset article numbers without clearing out the usenet load tables? I suggest you clear out your usenet_article, usenet_article_loader and usenet_ref tables and see if it happens again.

It will continue to loop as long as it finds articles that are ready for loading. You should see several of these 'loading replies' messages in sussession usually with decreasing numbers. It will then attempt to load the article into the forum. Whether it fails or succeeds to insert the post into the forum is irrelevant, it will be deleted from the loader table.

So there are only a few logical scenarios for this error to occur.

if the deletion failed for some reason. As you made no mention of an error message, the script may have been aborted at some point before the actual delete took place.
if the post it is trying to load actually originated from the forum in the first place and you have 'import existing posts' turned on, the article already exists in the post table and you reset the message counters to 0 at some point.
you changed the header information that identifies your site. The script has no idea that the post originated from you but it still found a postid and is trying to load it.

I will look at adding additional error checking to trap some of these.

[SNA]

Ok Cool...

Im using Cnews.. and it isnt adding the Lines: ## their from the posts coming off the Forums... Im guessing there is a config file somewhere to enable it...

Maybe you want to take that into condiseration...

If it doesnt find a Lines: then it should do something other consider it a binary..

Just a suggestion.. GREAT HACK :) It will be used.

[fastforward]

Originally posted by SNA
Ok Cool...

Im using Cnews.. and it isnt adding the Lines: ## their from the posts coming off the Forums... Im guessing there is a config file somewhere to enable it...

Maybe you want to take that into condiseration...

If it doesnt find a Lines: then it should do something other consider it a binary..

The only other way to determine if it's a binary post is by your own configurable spam filters. The obvious one to check for is 'begin 644'. This is actually added by default. The lines check is really only an additional check and a way for you limit the size of messages you want to download.

You say the server isn't adding the Lines field. As you mention, it is probably a configuration issue. What about other articles, or are you receiving a full IHAVE usenet feed? If it's just acting as a client to an upstream news server then you will probably see the Lines header on all other posts. If the issue is only with articles that are posted directly to your cnews server then there's not really a problem as when that message is propogated to another server, the field will be added.

The other point of course is why would we care if forum posts have it or not. They don't even get parsed by that code. We don't need to check them as we're not even downloading them. :)

[SNA]

OK I run my own newsgroups.. I am not a client.. I am a Server.

Lets say people post to the forums and I have to delete all the posts for some reason... when I go to download all the messages again it skips past all the ones created from the forums because there is no Lines: ## Line.

[Mark Hewitt]

Hey!! it works, and I didn't change anything either. Thanks fastforward for all your help. I must have got quite boring over the past couple of weeks.

Just a couple of questions on how posts appear on usenet.

I used quote to put the quoted text from the previous post in my reply and it formatted it correctly on usenet with the '>' characters in the correct place but even though I wrote my reply below the quoted text on usenet it appeared above it. On the newsgroups I posts to some people can get very upset about that.

Also
[ QUOTE ] [ /QUOTE ] (without the spaces) appeared at the top of the post.

[fastforward]

Originally posted by Mark Hewitt
Hey!! it works, and I didn't change anything either. Thanks fastforward for all your help. I must have got quite boring over the past couple of weeks.

Just a couple of questions on how posts appear on usenet.

I used quote to put the quoted text from the previous post in my reply and it formatted it correctly on usenet with the '>' characters in the correct place but even though I wrote my reply below the quoted text on usenet it appeared above it. On the newsgroups I posts to some people can get very upset about that.

Also
[ QUOTE ] [ /QUOTE ] (without the spaces) appeared at the top of the post.
I re-wrote all the outgouin post bits recently to try to tidy it up. I'll add an option for you to decide whether you want the quote above or below the message. The other thing with the '[ QUOTE ]' showing must be a bug. Does it happen all the time?

[Mark Hewitt]

Can't say if it happens all the time as I have only sent one post to usenet :)

I take it from this that it doesn't support multiple quotes within a single post?

[annams]

Not only for quote, it happens for URL too.

In case of URL it appears as [ url. The url and the closing [ /url ] are missing.

[fastforward]

It seems it depends on exactly what order the vb code appears in the message. It's not very easy to catch them all. Just taking the URL code as an example, I have to check for [ URL ], [ URL =.... ], [ URL ="..... ] etc. And then there is the problem of posts that embed the url by doing a please click HERE (http://vbulletin.com) type thing. I either replace the HERE with the underlying url which may make the sentence to appear nonsense, or I could simply strip out the URL which is probably even worse. Then there's quotes and embedded quotes, colors with the # colors without the # but with double quotes.... I could go on, but you get the picture. And if that weren't enough, you have custom bbcodes :(

Anyway, I'll re-visit the regexes that trap these and see what I can do. The safest solution, however, is to turn vbcode off for usenet forums.

[annams]

I actually tried turning off vB code for the forum, then it had USENET articles displayed as [ URL ] url goes here [ /URL ].

So I quickly turned it back on. Thanks for looking into it.

[fastforward]

Originally posted by annams
I actually tried turning off vB code for the forum, then it had USENET articles displayed as [ URL ] url goes here [ /URL ].

So I quickly turned it back on. Thanks for looking into it.
Yes, you'd need to change the options in the usenet control panel to stop urls and smilies being converted. Unfortunately, that won't fix historical posts as the bbcode is added before the post is inserted into the forum.

Not to worry, I'll at least sort out the URL, QUOTE and standard bbcode problems in the next release. Simple custom bbcodes with no parameters are easy to trap for aswell. It's just the custom codes with multiple params that will be virtually impossible to fix.

[annams]

Could you please tell me when I can expect the next release? Thanks.

[fastforward]

Originally posted by annams
Could you please tell me when I can expect the next release? Thanks.
Sunday probably. I have a fulltime time job and a family. This is already taking way too much of my time :p

Once all these little niggly bugs are fixed I'm going to slow down for a bit. I've also got to spend some time re-doing my sites when vb 2.0 is released.

But don't worry, I've still got plenty of features planned for this hack like support for binary groups and email to PM gateway so I'm not going to give up on it. Besides, I need it for my own sites.

[Mark Hewitt]

It's working great at the moment but I'll tell you how I would like to see it work, for you to consider in later versions.

The thing which concerns me at the moment is the way that the posts are appearing on usenet, namely the fact that they only appear in reply to the first post, this being a restriction of having a linear board.

However how I'd like it to work is to create a seperate template set for use in the newsgroup forums, in this the "Post Reply" button will be completely removed.

Instead underneath each individual post I would put a link "Reply to this post", as far as the forum goes this would do exactly the same thing as Quote does (thus forcing people to quote the text). However when the posts are sent out to usenet the posts appear threaded in the correct way with the posts appearing in their correct positions in the threads.

[fastforward]

I think I can do this relatively easily. It would mean a vB code edit in newreply.php and, as you mentioned, a complete new template set. Although it will still work without templates and would depend on whether you clicked the 'Reply' or the 'Quote' button.

I agree that it would be better this way. In fact I seriously considered switching to wwwThreads and write the hack for that board instead. It allows each site visitor to switch between in-line and threaded view. That way you could see the usenet posts in a threaded format too. The thing that stopped me was the lack flexible custom templates via templates.

Anyway, I'll see if I can get it working this weekend. I'll make it a global control panel option.

[annams]

Hi,

On the web server I use there are limits on the resource usage. Because of these my cron job that runs newnews.pl is getting killed quite often. My questions are:

1. When the job gets killed in the middle, would that leave either the newsgroups data or the forums data in any inconsistent state?

2. Can you modify the script to rotate through the newsgroups, so whatever newsgroup that is currently being fetched first would be fetched last in the next run. Or better yet give the option to do few newsgroups everytime.

On my web server the following are the resouce limits:

Size of Core Files - 0 MB
CPU Time Used - 30 seconds
Data Size - 3 MB
File Size Created - 1 MB
Memory Locked - 1 MB
Number of Open Files - 32
Number of Simultaneous Processes - 8

I appreciate your help (and the hack :-) )

[annams]

Does the news server need to be connected / disconnected for fetching data from every newsgroup? Can the same connection be reused for all the newsgroups currently being processed?

Please read the above question in light of resource usage problems I am having.

[fastforward]

Originally posted by annams
Does the news server need to be connected / disconnected for fetching data from every newsgroup? Can the same connection be reused for all the newsgroups currently being processed?

Please read the above question in light of resource usage problems I am having.
The script will only reconnect for each group if you have the 'Multiple Servers' option enabled. This was the the very reason I made that an option. If it's set to No, the script will make one and only one connection for the entire run.

It may be too difficult to get the script to remember where the it died on the previous run and adjust the order accordingly, but adding an option to limit the number of groups may be doable. However, the option is already there (sort of). All you need to do is adjust the batch size option. This was why the option was included. By usung this method, you can guarantee that at least some messages will be pulled from each group and that all outgoing messages will be processed.

[annams]

Yes, I had the multiple servers option, while disabled the script runs much faster. Thank you.

[jarvis]

Okay, I've searched thru most of the posts on this great hack, but I may have skipped over, or it just isn't there.

I'm not a technical expert, but would like to think I can tackle this task. Has anyone implemented this on a Win32 system, and if so, did you use the Active Perl install? Does the ActivePerl install have all of the needed mods required for this? Lastly, do I have to install an ISAPI filter thru the IIS MMC, 'cuz right now all I see is the PHP filter listed there.

Thanks, and don't laugh too hard. :)

[fastforward]

Originally posted by jarvis
Okay, I've searched thru most of the posts on this great hack, but I may have skipped over, or it just isn't there.

I'm not a technical expert, but would like to think I can tackle this task. Has anyone implemented this on a Win32 system, and if so, did you use the Active Perl install? Does the ActivePerl install have all of the needed mods required for this? Lastly, do I have to install an ISAPI filter thru the IIS MMC, 'cuz right now all I see is the PHP filter listed there.

Thanks, and don't laugh too hard. :)
I wondered how long it would be before someone wanted to run this on NT :)

The short answer is, it might work.

The latest Active state ports provide just about all the Unix functions (even fork) so there shouldn't be any problems there. The only thing I know of that Windows doesn't support is the ALRM system calls. I haven't used that so it doesn't matter.

However, you may have problems with some of the modules that are required for the script to run. I know for a fact that Active State doesn't provide binaries for all of them. If you have MSVC 5 or higher you should be able to compile all the modules just as you do for a Unix installation. You just substitute 'make' with 'nmake'.

Good luck and let me know if you have any success :)

PS. I know nuffink about the PHP/ISAPI stuff so you're on you own there :)

[n3n]

replies to a post not working..

i see the first post from usenet but the child replies do not seem to be posted on usenet.. -.-;;

[fastforward]

Originally posted by n3n
replies to a post not working..

i see the first post from usenet but the child replies do not seem to be posted on usenet.. -.-;;
My guess is this:

You started a thread locally, then you replied to it a couple of times, also locally. Then you ran the script once and only once.

Run it again. The replies can only be sent when it's known what the message id is of the post it's replying to. As the thread starter hadn't been sent to usenet it doesn't have a message id.

[fastforward]

Just a quick update on how the next release is going for those interested.

I've managed to get it working so that you can reply to individual posts within a thread and have it show up correctly threaded on usenet. It requires one single code edit in newreply.php and a hidden input field in the newreply template.

I've also fixed the stripping of bbcode problems... I think. The only way to really test it is over time.

There were also a few other little things that I fixed including SNA's request for a workaround for his CNEWS set up.

I need a bit longer to test it but I'll be putting up a new version of dBforums.com tomorrow with a complete new set of templates for the usenet forums to better accomodate usenet posts.

If you want anything else (anything easy) included in the next release speak now or forever hold your peace :)

[tamarian]

Originally posted by fastforward


If you want anything else (anything easy) included in the next release speak now or forever hold your peace :)

How about a a listing of the database schema? Only the relevant table/fields.

I'm still running the old hack 1.8 on RC3 since my database is no longer compatible with the last hack. Couldn't figure which fields I didn't revert/update.

Looking forward to the update!

[Gilby]

Originally posted by tamarian
How about a a listing of the database schema? Only the relevant table/fields.

I'm still running the old hack 1.8 on RC3 since my database is no longer compatible with the last hack. Couldn't figure which fields I didn't revert/update.

Looking forward to the update!

Yeah, that would be nice... a script that will check to see if it set right and if it's not, then it changes it to the right thing would be nice. But let's see the next version first (the bbcode bugs are annoying).

[fastforward]

Gilby,

If you want to try out the new function to strip the bbcode, replace your existing one with this.

You can be the guinea pig and take all the flack from the usenet users if it goes wrong :D No, but seriously, I'd appreciate some help in testing it. It's getting a bit boring posting test messages to myself :)

sub remove_bb_code {
my $text = $_[0];
my ($bbo,$bbc);
$text =~ s/&amp;/&/ig;
$text =~ s/\[size=[0-9]+\]|\[\/size\]//ig;
$text =~ s/\[color=(\"\#)?[A-Za-z0-9]+(\")?\]|\[\/color\]//ig;
$text =~ s/\[url=(\")?//ig; $text =~ s/(\")?\](.)+\[\/url\]//gi;
$text =~ s/\[email=(\")?//ig; $text =~ s/(\")?\](.)+\[\/email\]//gi;
$text =~ s/\[font=(\"\#)?[A-Za-z]+(\")?\]|\[\/font\]//ig;
$text =~ s/\[list(=)?[1Aa]?\]|\[\/list(=)?[1Aa]?\]//ig;
$text =~ s/\[\*\]/ - /ig;
$text =~ s/\[(\/)?code\]//ig;
my $bbcodes = db_fetch("SELECT bbcodetag FROM bbcode");
while (my $bbcode = $bbcodes->fetchrow_array) {
$bbo=quotemeta("[".$bbcode."]");
$bbc=quotemeta("[/".$bbcode."]");
$text =~ s/$bbo|$bbc//gi;
}
return $text;
}

[Dioxin]

Originally posted by tamarian


How about a a listing of the database schema? Only the relevant table/fields.

I'm still running the old hack 1.8 on RC3 since my database is no longer compatible with the last hack. Couldn't figure which fields I didn't revert/update.

Looking forward to the update!

this would be a nice feature. i am also on hack release 1.8 and i plan to alter the tables manually. this would save me a lot of time :)

dioxin

[jarvis]

Well, update on the Win32 install...

Looks like this won't work with Win32 systems without additional coding.

Also, some of the Perl mods don't match up, or I can't find them.

Net::NNTP and SMTP are not listed as ActivePerl mods, but NNTPClient and SMTP client are. Might be same, but then again, I'm not a technical expert.

All of the others I can find, except Date::Parse and MIME::WordDecoder.

So, I guess I'll have to wait until someone ports this to a Win32 Environment (hint, hint, maybe even money) :D

Thanks though for the hack...

[n3n]

here is html version of it.. nothing that i have done
i just added content type and br tags to it
so you can execute it online with browsers..

i was tired of getting into telnet and execute it to see the realtime result of the page.. :mad:

just replace the existing newnews.pl with attached file (newnews.txt) and rename it to newnews.pl and call it from a web browser..
i checked if it works with crontab and it works of course.. ;)

[fastforward]

Nice one. I'll see if I can get time to add this. I'll add some checking to see if it's running on the console or a browser and hide the <br>'s accordingly.

[SeanM]

Are there any problems with upgrading to RC3? I have noticed that the Usenet Gateway hack has been updated to work with RC3. Just curious if there's any problems..

[n3n]

Originally posted by fastforward
Nice one. I'll see if I can get time to add this. I'll add some checking to see if it's running on the console or a browser and hide the <br>'s accordingly.

Great!!
(so i don't have modify coming up version of it any more..:cool: )

[Dioxin]

one question:

if i add this:

// START USENET HACK
ORDER BY seq, dateline $postorder
// END USENET HACK

to showthread, the postorder istn't correct. i use the latest release.

why do you use this function in newnews.pl:

sub order_local_posts {
my $posts = db_fetch("SELECT threadid,postid,dateline FROM post WHERE isusenetpost=0 AND ord=0");
while (my $post = $posts->fetchrow_hashref()) {
my $countposts = db_fetch("SELECT COUNT(*) FROM post WHERE threadid = $post->{threadid} AND dateline < $post->{dateline}");
my $ord = $countposts->fetchrow_array;
$countposts->finish;
db_execute("UPDATE post SET ord=$ord, seq=$ord+1 WHERE postid = $post->{postid}");
} $posts->finish;
}


it counts all posts and update the seq-field. but wgy?? if a member post to a forum, the post is added with seq = 0 and the post is displayed in the middle of the thread. so i removed the "order by seq".

please help, thanx in advance.

[fastforward]

The date you have elected to use for posts in your control panel will determine whether these functions are needed.

If you remove the order by seq line and you have selected nntpdate in your control panel, then usenet posts will not be ordered correctly for reasons explained in several posts in this thread. To get around it the posts are 'sequenced' and the thread is ordered using that column. The order_local_posts() function is to allow people to use the nntp date without editing the newreply.php file. It will simply add the sequence number for any local posts when the script is executed.

This is all irrelevent in the next release anyway so I shouldn't worry about it to much. But the bottom line is in the current version, those things are needed unless you want confused users.

[Mark Hewitt]

I actually removed the seq part a long time ago as it was causing threads in my normal forums to appear out of order.

[Dioxin]

for your "fast" answer :)

when is the next release expected?

[fastforward]

Originally posted by Dioxin
for your "fast" answer :)

when is the next release expected?
Definately this weekend... I promise :)

I have it all up and running at dbforums.com, I just need to document it all. There's been a few changes in the tables and code edits required.

To get the most out of this next release, you need to create a seperate template set more suited to a usenet article layout. I decided it was silly trying to squeeze a usenet article into the forum layout when I should be changing the forum layout to accomodate the usenet article.

Another change (related to your seq issue) is that both local timestamp and nntp timestamps are stored in the table. The seq column has been removed. You now just order on dateline (default) or the nntpdateline.

[fastforward]

And here's another one...

This is release 20010519 for vB2.0 RC3.

Quite a lot of changes in this one. The biggest change is probably the ability to reply to individual usenet posts at any level of the thread and have ot thread correctly on usenet. (The quote button is used for this.) For this to work you MUST make a small code edit to newreply.php AND add a hidden field to the newreply template.

To get the most out of this you should consider creating a complete set of custom templates for the usenet forums. You can then show the nntpdate but still maintain a logical order by dateline. You can also hide the posts/location etc and display the full nntp From: header just like a real usenet post. See http://dbforums.com to see what I mean.

Changes

Added reply to particular post capability to ensure outgoing posts are threaded correctly when they reach usenet.
Changed indexing routine to support the new vB search thread title option.
Changed method of ordering usenet posts in forums to a more robust method. (Additional column containing nntpdate replaces the seq column)
Improved stripping of vB codes from outgoing posts.
Added new column containing nntp from line that can be referenced in templates.
Added global 'posting enabled' switch to control panel.
Improved autopurge function. It's now a lot quicker.

Bug Fixes

Fixed problem of lastactivethread not being updated when a post was deleted.
Fixed problem with mailing archives threading. Some posts were not being loaded if there were no references.
Fixed bug causing quotes not to be indented.
Probably some other stuff that I can't remember.

If possible you should do a clean install with this version. If not possible, you should go through the table structures carefully. Let me know if you need clarification on the need of any columns.

With only days before the vB final (hopefully) I want to clear up any bugs asap. If you do find any, make sure they're little ones that can be fixed easily :)

Download (http://britishexpats.com/download/usenet_gateway.tar.gz)


Note to n3n: I didn't get around to adding the html to newnews.pl. I decided to wait and do it properly with an option to run it from the control panel and automatically set up a cron job etc.

Note to jarvis: I'll see what I can do about getting it to work under windows. It does work under Cygwin though if you can't wait.

[jarvis]

Fast,

You've got an awesome product, and if you are doing dev on a Win32 version, then I will wait. Like I said b4, I am willing to pay for this version as it is exactly what I've been looking for. Right now, I'm using DNews from netwinsite, but if I could incorporate ng's into my existing forums, whoa... (drool) Darn you Unix giys :)

[Dioxin]

hi fastforward,

i get these message from my telnet:

Query failed:
UPDATE usenet_outgoing SET refs = concat(refs,' ','<3b08a44a$0$304$6e37e724@news.dpn.de>') WHERE inreplyto = 23810

DBD::mysql::db do failed: Unknown column 'inreplyto' in 'where clause' at newnews.pl line 601.

the table usenet_outgoing has no field called 'inreplyto'

please help.. i use the latest version.

btw, your instruction set for the new release is not complete. all the insert statements for the spam- and usenet_replace table are missing.

[fastforward]

Originally posted by Dioxin
hi fastforward,

i get these message from my telnet:

Query failed:
UPDATE usenet_outgoing SET refs = concat(refs,' ','<3b08a44a$0$304$6e37e724@news.dpn.de>') WHERE inreplyto = 23810

DBD::mysql::db do failed: Unknown column 'inreplyto' in 'where clause' at newnews.pl line 601.

the table usenet_outgoing has no field called 'inreplyto'

please help.. i use the latest version.

btw, your instruction set for the new release is not complete. all the insert statements for the spam- and usenet_replace table are missing.
Yeah I just got those errors aswell. I forgot to add the inreplyto columns to the usenet_outgoing and the usenet_outgoing_log table. Outgoing messages will not be sent until I fix it. You can add the columns yourself if you like they should be of type int and unsigned. I can't get a fix out until tomorrow night I'm afraid. I also need to make sure there are no more issues with the release version.

Sorry about that.

[chrispadfield]

Now vb2.0 is out i am really looking forward to this.

I do have one request though, we talked about this before a bit.

Basically, the "first time" a user goes to post a newsgroup message i want a confirmation page to come up explaning exactly what they are doing.

What would be needed here..

i) an extra field in user Y, N for has seen warning sort of thing.

ii) an extra modification to newthread/newreply.php to check for Y/N if it is a newsgroup post. If N then display a new page with warning on it, with the message etc as hidden variables. If they press "accept" or whatever you use then the message is sent and everyone is happy.

i would really like this if it is possible.

[fastforward]

Originally posted by chrispadfield
Basically, the "first time" a user goes to post a newsgroup message i want a confirmation page to come up explaning exactly what they are doing.
The way I have done this on dbforums.com is using templates. Every usenet post has a sidebar where all the vbcode stuff used to be advising them that their post is about to placed in the public domain and to read the faq etc etc...

If you really really really really need it :) I'll put it on the to do list, but I desperately need to spend some time getting my site updated. My intention was to fix any bugs and leave it alone for a few weeks.

There's still plenty I want to add to the hack in the next big release though, including full mime and binary attachment support.

[fastforward]

This release certifies the hack for vB 2.0 :)

There have been a couple of bugfixes:

Added missing columns to usenet_outgoing & usenet_outgoing_log tables
Fixed problem with autopurge. Local posts that were made in a usenet forum were being expired too early... much much too early!


Download (http://britishexpats.com/download/usenet_gateway.tar.gz) latest version for vB 2.0

I was going to officially move the hack to vbulletin.org but the discussion bit is broken so it'll have to stay here.

[chrispadfield]

i like how you have done it, update your site i will work on a new tmeplate set, seems a good plan with the new reply feature as well.

[mcncyo]

the only thing i can't get installed with CPAN is the Bundle::MySQL i went to CPAN website and it isn't listed anymore.

Does anyone know where i can get it?

[fastforward]

Originally posted by cyo
the only thing i can't get installed with CPAN is the Bundle::MySQL i went to CPAN website and it isn't listed anymore.

Does anyone know where i can get it?
It's actually part of 'Msql-Mysql-modules-1.2216' They must have changed the name on me :)

Here's the link: http://search.cpan.org/search?dist=Msql-Mysql-modules

[annams]

I am using the default template and have BB code turned on in the forums. Whenever a new post is made while keeping the "Automatically parse URLs" turned on, the resulting post shows up with the following text in the usenet:

[ url

No url is sent to the usenet posting. This was a problem present in the older version. Can you please provide a fix.

Thanks.

[Gilby]

Originally posted by annams
No url is sent to the usenet posting. This was a problem present in the older version. Can you please provide a fix.

Here is a fix (within the remove_bb_code subroutine), replace this:

$text =~ s/\(.)+\[\/url\]//gi; # url
$text =~ s/\[email=(\")?//ig; $text =~ s/(\")?\](.)+\[\/email\]//gi; # email

with this:

$text =~ s/\[url(=)?(\")?//ig; $text =~ s/(\")?\](.+)\[\/url\]/ $2/gi; # url
$text =~ s/\[email(=)?(\")?//ig; $text =~ s/(\")?\](.+)\[\/email\]/ $2/gi; # email


The probelm was that it didn't recognize the url code when it was like: [ url ] http://www.gilby.com ((\")?//ig; $text =~ s/(\")?\) [ /url ]

I also made it so that any internal text that is in the url code is not removed, so if I had [ url="http://www.gilby.com" ] My Site [ /url ], it would replace it with [url]http://www.gilby.com My Site. Still not perfect, but no text is lost.

[fastforward]

Thanks Gilby. I wasn't looking forward to trying to fix that one :)

[AWS]

I got this hack running on Win2k and Apache. Check it out at http://resource-zone.com

If any else wants to get it running with NT I'll post a howto to this thread. The way I set it up should also work with IIS.
Bob

[jarvis]

Ooh, ooh.... me, me!!

I want an NT/IIS how-to!! :D

I gave it an initial effort, but realized I was in over my head, and I couldn't translate some of the needed modules over to ActiveState Perl. Would love to try your suggestions.

Thanks!

[NgtCrwlr]

fastforward, words just can't tell you how much I appreciate your vBulletin usenet hack. I had installed the hack on rc2 dev board just to see if I could get all the perl modules installed etc. The rest is history... I am now working on customizing the vB2 final usenet hack.

I'm wondering if you are planning to release your usenet templates? The templates are my next project and if you don't mind... being able to use your templates as a guide would sure save time and a few brain cells.

Thanks again for sharing your brilliant work.

Cheers!

Larry "NgtCrwlr" Mingus
http://www.makeitsimple.com

PS: You should hook up with a hosting company because anyone using your hack will soon need a dedicated server. :D

[fastforward]

Ah.. kind words indeed. It makes it all worthwhile :)

I can't take all the credit though. My thanks go to:

Gilby for his code fixes and orphan handling functions.
Chris Padfield for keeping me motivated early on and providing the feature list.
Tamarian for not suing after trashing his forums ;)
Anybody else brave enough to install this hack on their production server :eek:
Jelsoft for ummm... that forum software thingy they wrote :p
And most importantly, Randall Schwarz for Perl and all the Perl module writers... Yayyy! :D


PS I've had a few requests for the templates so I'll make the them available tomorrow.

[Dontom]

Hi, I am trying to install this hack - does this mean Date/Parse.pm is not installed?

TIA
:D
Tom


Can't locate Date/Parse.pm in @INC (@INC contains: /usr/lib/perl5/5.6.0/i386-lin
ux /usr/lib/perl5/5.6.0 /usr/lib/perl5/site_perl/5.6.0/i386-linux /usr/lib/perl5
/site_perl/5.6.0 /usr/lib/perl5/site_perl .) at ./newnews.pl line 48.
BEGIN failed--compilation aborted at ./newnews.pl line 48.

[Gilby]

Originally posted by Dontom
Hi, I am trying to install this hack - does this mean Date/Parse.pm is not installed?


Yup, that error means that this perl module is not installed. Look in the installation instructrions of this hack which explains how to install the required modules.

[Dontom]

Sorry, again a dumb perl-question..
I have just installed above mentioned module, now I get the following (I uploaded newnews.pl ascii and chmoded it to 755)

sh: /bin/ps: Permission denied
Killed
Any Idea?
Tom

[Gilby]

Originally posted by Dontom
Any Idea?

Comment out this section:
# check for previous instance
my $pid=$$;
my $old_pid=0;
if (-e $pid_file) {
open(W,"$pid_file");
$old_pid=<W>;
close(W);
}
open(W,">$pid_file");
print W $pid;
close(W);
my $ps_pid=`/bin/ps xaww | grep newnews.pl | grep -v "grep"`;
$pid=substr($ps_pid,0,5);
if($pid+0 == $old_pid+0){
system("kill -9 $old_pid");
}

Just put in # at the beginning of each of those lines. That code checks to see if the script is already running and if it is, it terminates itself. Apparently the command to check if it's running is not set to give you permissions to run it on your server.

[AWS]

How I installed on win2k.

Lets get the tool you'll need first.

Download nmake.

ftp://ftp.microsoft.com/Softlib/MSLFILES/nmake15.exe

Log on as Admin.

Create a dir on C. Mine is called mods and extract nmake there.

Open a console and follow the directions in the usenet install file to install the modules.
When you run CPAN the first time it will ask you to configure it. The main paths you have to set is the dir to download to which would be the dir you created.
The path to nmake.
Path to your zip utility.
Make sure you set to ask before it installs dependencies.
The rest you don't have to worry about.
3 of the modules will install with no errors 2 will build but won't install, but, we won't let that stop you.
During the install it will find some dependencies. Let it install them but remember the ones that error. One of the 3 errored on my install.
You'll know when they error.
Now you have all the modules you need.
For the ones that errored we''ll just move the files to where they need to be.
I don't remember which ones errored but to move them is pretty easy.
After you have all the modules installed. exit CPAN and command prompt cd to the dir you extracted the newnews.pl file. Once there type

perl newnews.pl

it will error but we want it too so we can see where to move the files to.

When it error you get a can't find dirname/module.pm with the path to all the perl modules. Dirname and .pm name is what we want. Mail was one that errored on mine. What you'll do is open explorer and head into C:\path\to\perl\lib dir. If you don't have the dir name it was looking for create it and then head into the build directory you setup. You will see dirs for each modules you created. Open the mail dir and in it you'll find a folder called blib open it and you'll see another folder called Mail. Open it and copy the .pm file into the mail dir you created in /perl/lib.
If you get an error can't find module.pm just copy the module out of the build dir into /lib.
Run the script again and watch for the next error. Do the same thing you did above for each error.
When you get to the error 'kill not a recognized command or batch file' all the modules are installed and working correctly.

Now the last step.

Go to

http://sources.redhat.com/cygwin

and download and install the gnu tools. You need them for the script to run. Do the net install. Install them all. This is about 16 megs of files. When it starts the install choose to install for all. That way all users can use them.
When they are installed click the cygwin icon on the desktop to get to open the bash shell.

cd /path/to/newnews.pl

then ./newnews.pl

perl will overload and you'll get a stackdump error. No problem. We only wanted to let windows know cygwin is installed. If you happen to get any module errors there don't worry about them.
Exit the shell and open windows console again. Make sure you create the forum to hold the postings before you run the script.

cd path/to/newnews.pl

then perl newnews.pl

Watch the magic happen as it grabs the news.
This will take about any hour. The install time I mean not the time it takes to get the news.
I'm sorry I couldn't be more specific about the exact files I had to move and the dirs I had to create.

Fastforward, thanks for the great hack. I should have never switched to win2k from FreeBSD. At least they let me keep Apache. Wouldn't have to go through all the trouble to get the hack running.

Bob

[Dontom]

Gilby, Thank you - it works!!!
:D
Tom

Originally posted by Gilby


Just put in # at the beginning of each of those lines. That code checks to see if the script is already running and if it is, it terminates itself. Apparently the command to check if it's running is not set to give you permissions to run it on your server.

[NgtCrwlr]

Those of you trying to get the proper modules installed you may find this little program handy. I found this at Gossamer Threads about two years ago, knowing that some day I might need it.

Then along comes fastforward with his modules dependent marvelous Perl wizardry, and I just smiled and said "I think I can handle this, now where did I put that little program?." ;)

Save this code as a .cgi file (modules.cgi or whatever) then upload to your cgi-bin and call it with your browser.

Example:
http://www.mis-forums.com/cgi-bin/modules.cgi


#!/usr/bin/perl

# uncomment the line below if you have installed any modules locally
# change the line to reflect your server settings

# use lib "/path/to/my_modules/lib/perl5/site_perl/5.005";

use strict;
use File::Find;
my (@mod, %done, $dir);
find(\&get, grep { -r and -d } @INC);
@mod = grep(!$done{$_}++, @mod);
foreach $dir (sort { length $b <=> length $a } @INC) {
foreach (@mod) { next if s,^\Q$dir,,; }
}
print "Content-type: text/html\n\n";
foreach (@mod) { s,^/(.*)\.pm$,$1,; s,/,::,g; print "$_<br>\n"; }
print "Done! ($#mod modules!)\n\n";
sub get { /^.*\.pm$/ && /$ARGV[0]/i && push @mod, $File::Find::name; }


Cheers!
-Larry

[fastforward]

A few people have asked for a copy of my templates to use as an example for the usenet bits. Here's the default and usenet styles from dbforums.com

They were actually created on RC3 so there are some that should really be reverted. I was going to tidy them up but other peoples HTML skills are probably a lot better than mine... besides, Startrek Voyager two hour 'end of series' finale is on tonight so I don't want to be messing about with templates :D

[chrispadfield]

Originally posted by fastforward


They were actually created on RC3 so there are some that should really be reverted. I was going to tidy them up but other peoples HTML skills are probably a lot better than mine... besides, Startrek Voyager two hour 'end of series' finale is on tonight so I don't want to be messing about with templates :D

www.talkstartrek.com wants to hear your comments afterwards ;)

[NgtCrwlr]

fastforward... a big high five at-chya, I managed to get the templates working well enough that I can go to bed with a gleaming smile.

http://www.mis-forums.com/vbbs/

Thanks man, I hope some day to be able to repay you in some way, for the work you have put into this project. This is just too good to be true. :cool:

That's it I'm down for the count.

Cheers!
-Larry

[NgtCrwlr]

I think I have found a posting date bug....

I finally got around to setting up a test forum to try out posting from vB2 to usenet. The first two posts one a reply and the other a new thread are both showing Date: 12-31-1969 07:00 PM in the usenet template.

Here's what it looks like.
http://www.mis-forums.com/vbbs/showthread.php?threadid=9445
http://www.mis-forums.com/vbbs/showthread.php?threadid=9413

From phpmyadmin I see that the nntpdateline field is 0 for both posts, so I'm off to figure out why it is not being updated.

If anyone has already figured this out I'd appreciate a toot.

Cheers!
-Larry

[fastforward]

I'm assuming it's not happening on all posts right?

My guess is that for whatever reason, it couldn't find the date in the article header. Before the nntppostingdate column was introduced the script used to put the date straight into the dateline column. This dateline value was checked and set to the current date if null. This extra error checking seems to have been forgotten this time.

However, it doesn't explain why there are some articles coming through that the script can't parse. I've not actually seen it happen on my setup, so my guess is it's specific to a newserver and not an article. I did see it now and again when I was using Newsfeeds.com as a provider.

Let me know if you find anything out. :)

[fastforward]

I've just had another thought about this. The only other thing different between instalations that would affect the date is the version of the Date::Parse module. I use the str2time function from this module to parse the date and convert it into unix time.

There is a similar version number incompatibility with the DBI module and the 'mysql_insertid' function. In the older version, the function was returning null. I was pulling my hair out for weeks over this one as only captnroger had the problem. Then just last week, eric figured it out when he encountered the problem.

[NgtCrwlr]

I think... I found it.

The problem was not posts coming from usenet, it's on the local end when I replied to or started a new thread. What I did was add what I think is missing nntpdateline updates:

newthread.php
$DB_site->query("INSERT INTO post (postid,threadid, ..snipped.. ,iconid,visible,nntpdateline) VALUES (NULL,'$threadid',' ..snipped... ,'$iconid','1','".time()."')");

newreply.php.
$DB_site->query("INSERT INTO post (postid,threadid,title, ...snipped... ,iconid,visible,nntpdateline,inreplyto) VALUES (NULL,'$threadid','".addslashes(htmlspecialchars($title))."', ....snipped...,'$iconid','$visible','".time()."','$inreplyto')");

Unless I'm missing something I think that's it? At least it seems to work right so far. I'm still waiting on usenet propagation to see what they look like externally.

[fastforward]

You're right. I was just about to say it's in the instructions as a necessary code change, but I just went and checked and it's not there. That's exactly what I have on my setup though, so I guess I forgot to put it in the instructions. ooops :o

I intend to put up another little release tomorrow to include gilbys fix so I'll fix the instructions at the same time.

I've also got another little option that allows you to reduce the font size of quotes. I think it makes the posts easier to read. I've put it on dbforums.com already so you can see what I mean.

[NgtCrwlr]

The quote option sounds great. :)

One other tiny problem you may want to fix is in newnews.pl
When doing a history pull and all articles on a run are not available last msg: counter does not increment and has to be updated by hand.

Also, it would be nice if when there are no messages in a group that the program would skip checking orphans and jump right to the next group. With a lot of groups I think that little diddy would speed things a bit.

Don't want to load you down again Paul, I just thought you might want to know about these being that you are updating again so soon.

Cheers!
-Larry

[mcncyo]

I am getting this problem


Query failed:
INSERT LOW_PRIORITY INTO post (title,allowsmilie,threadid,dateline,nntpdateline, pagetext,visible
,msgid,ord,username,userid,nntpfrom,ipaddress,isus enetpost) VALUES ('What is required for salvat
ion?',1,,990971890,990207918,'What do I need to do, be or whatever to be "saved"?\n','1','<3B055
FD2.88A00F0C@hotmail.com>','0','Tim',0,'Tim #timothyb24@hotmail.com#',0,1)

DBD::mysql::db do failed: You have an error in your SQL syntax near '990971890,990207918,'What d
o I need to do, be or whatever to be "saved"?\n','1',' at line 1 at ./newnews.pl line 609.






[QUOTE]

[fastforward]

Originally posted by cyo
I am getting this problem
....

update your DBI module. The version you have is not returning the threadid.

[fastforward]

Originally posted by NgtCrwlr
The quote option sounds great. :)

One other tiny problem you may want to fix is in newnews.pl
When doing a history pull and all articles on a run are not available last msg: counter does not increment and has to be updated by hand.
I think someone else had this problem and I couldn'd understand what they meant. However, I switched news serrvers this weekend and re-pulled all news and I saw the problem with my own eyes. I'll get on it.

I hope to get all these fixes done today and a new release out. If not today, definately tomorrow.

[NgtCrwlr]

While I was validating the output HTML I found that signatures from usenet messages can have invalid characters like (&, and high ASCII chars). htmlentities() action needed?

Also, I would like to see a nl2br in usenet signatures.

Oh-oh I think I hooked a big one!

I knew something wasn't quite right with... function update_last_active_thread because I have one thread title that displays B& amp;W Album. Looking into it further I think I see what is happening.

In function update_last_active_thread you are using
addslashes(htmlspecialchars($lastactivethread))
Unfortunately htmlspecialchars is not needed because vB has already preformed special character conversion and stores it converted in the DB.

This caused me to look further and I found that newnews.pl saves the thread title to the DB without processing for special characters. That would be OK if vB did it that way too.

Paul, tell me I'm wrong... please! If I'm not wrong this means a total purge of all usenet posts and new history pull. This is not that big of a deal for me because I don't have many posting users. Although the news feed I use is terribly slow. :(

God I love this stuff! :D

Cheers!
-Larry

[slinky]

The download link disappeared. In fact, so did the site. Could be down for maintenance but I'd love to get hold of this hack and implement on my system.

[fastforward]

The site could be a bit dodgy over the next few days as I've just updated the DNS and moved it to a new server. Keep trying and you should get it... if I've set up all the redirects correctly :(

[tamarian]

Originally posted by slinky
The download link disappeared. In fact, so did the site. Could be down for maintenance but I'd love to get hold of this hack and implement on my system.

Hmmm, I think Fast forward's site (not dbforum, BritishExpats) is undergoing DNS propagation.. The old site redirects to an IP and the IP redirects to the domain name. Hope things are o.k. Fastforward, this may take a few hours to clear-up.

I've been there before. :)

[NgtCrwlr]

I just talked with fastforward tonight and he is working on an update that should be ready tomorrow or the next day.

You can disregard my posts above about finding a big error, I wasn't really in sharp mode when I posted those. There will be no need to do a new history pull with the updated version. The posts that are affected by the above problem will cycle out of the database quickly enough to not be a bother.

slinky, your gonna love this hack. :D

Cheers!
-Larry

[NgtCrwlr]

Shezz, I think the three of us should play the LOTTO today. Check out the post time on the last three posts. :)

[tamarian]

Originally posted by NgtCrwlr
Shezz, I think the three of us should play the LOTTO today. Check out the post time on the last three posts. :)

Yes, I figured the DNS thing at the same time fastward was posting about it!

O.k. what numbers are we picking? :)

[tamarian]

Originally posted by fastforward
if I've set up all the redirects correctly :(

I think some ISPs will get circular redirects (infinite loops) for a while :(

[slinky]

Originally posted by NgtCrwlr
I just talked with fastforward tonight and he is working on an update that should be ready tomorrow or the next day....
slinky, your gonna love this hack. :D
Cheers!
-Larry

Heheh, can't wait. I set up a usenet passthrough into a graphical BBS back in 1992 and haven't done so since. (I'm still ROFL from the lotto comment.)

It seems that now I'm being directed to the right site as expats.com is up (nice work). Problem is that it seems that there is no download directory and with bad addresses I get redirected to the home page.

[NgtCrwlr]

slinky, I feel your anticipation. Here's a gift to help tide you over until Paul finishes the update. This will give you a leg up by giving you time to get all the perl modules talking to newnews.pl. You don't have to worry about doing things twice, the new version should drop in with just a few minor changes. Just don't come crying to me if I'm wrong. ;)

This is version 2.0 for vB 2.0 final.
http://www.makeitsimple.com/download.php

Cheers!
-Larry

PS: I'm playing 3,6,9 tomorrow. :)

[slinky]

Originally posted by NgtCrwlr
slinky, I feel your anticipation. Here's a gift to help tide you over until Paul finishes the update. This will give you a leg up by giving you time to get all the perl modules talking to newnews.pl. You don't have to worry about doing things twice, the new version should drop in with just a few minor changes. Just don't come crying to me if I'm wrong. ;) Ooooooh.... many thanks for helping me with my condition of instant vb hack gratification! :) I accept your disclaimer and let the hacker beware! Good luck with the lottery. :D

[fastforward]

This release fixes a few bugs and adds one new option.

Made use of HTML::Entities to strip usenet post subjects of HTML characters prior to inserting into database. This is consistent with the way vB does it.
Fixed last message number updating when no posts are found during a batch run.
Added option to size usenet quote text.
Another fix to the remove bbcode function.

Upgrading

Replace newnews.pl
Install the HTML::Entities Perl module from CPAN if necessary. This is part of the HTML-Parser bundle and is most likely already installed.
Check the optional code change number 4 and change if necessary.
Run the SQL statements in UPGRADE.txt. This will erase your settings so make a note of your 'Useragent' setting so that you can reset it to the same value after the upgrade. You spam and replacement filters will NOT be touched.

Download version 20010528 (http://britishexpats.com/download/usenet_gateway.tar.gz)

[slinky]

Wow... just looking at the cp impressed me! But nuts.... I'm getting a lot of errors trying to install the modules. :( Tried to find some of the through CPAN.org and was having a tough time.... any ideas?



cpan> install DBI
Running make for T/TI/TIMB/DBI-1.15.tar.gz
Unwrapped into directory yes/build/TIMB000

CPAN.pm: Going to build T/TI/TIMB/DBI-1.15.tar.gz

Couldn't chdir yes/build/TIMB000: No such file or directory at (eval 26) line 42
19
======================================

cpan>install Bundle::MySQL
Can't install Bundle::MySQL, don't have an associated bundle file. :-(
at (eval 26) line 1806



===============================


cpan> install Text::Autoformat
cpan> install Date::Parse
cpan> install Email::Find
cpan> install URI::Find
cpan> install MIME::WordDecoder
cpan> install HTML::Entities

yields these errors:


Running make for D/DC/DCONWAY/Text-Autoformat-1.04.tar.gz
yes/sources/authors/id/D/DC/DCONWAY/Text-Autoformat-1.04.tar.gz: No such file or
directory
Checksum mismatch for distribution file. Please investigate.

Distribution id = D/DC/DCONWAY/Text-Autoformat-1.04.tar.gz
CALLED_FOR Text::Autoformat
CONTAINSMODS Text::Autoformat
CPAN_USERID DCONWAY (Damian Conway <damian@conway.org>)
MD5_STATUS
localfile yes/sources/authors/id/D/DC/DCONWAY/Text-Autoformat-1.04.tar.gz

I'd recommend removing
yes/sources/authors/id/D/DC/DCONWAY/Text-Autoformat-1.04.tar.gz. It seems
to
be a bogus file. Maybe you have configured your `urllist' with a
bad URL. Please check this array with `o conf urllist', and
retry.

Fetching with LWP:
ftp://ftp.perl.org/pub/CPAN/authors/id/D/DC/DCONWAY/CHECKSUMS
yes/sources/authors/id/D/DC/DCONWAY/Text-Autoformat-1.04.tar.gz: No such file or
directory
Checksum mismatch for distribution file. Please investigate.

Distribution id = D/DC/DCONWAY/Text-Autoformat-1.04.tar.gz
CALLED_FOR Text::Autoformat
CONTAINSMODS Text::Autoformat
CPAN_USERID DCONWAY (Damian Conway <damian@conway.org>)
MD5_STATUS
localfile yes/sources/authors/id/D/DC/DCONWAY/Text-Autoformat-1.04.tar.gz

I'd recommend removing
yes/sources/authors/id/D/DC/DCONWAY/Text-Autoformat-1.04.tar.gz. It seems
to
be a bogus file. Maybe you have configured your `urllist' with a
bad URL. Please check this array with `o conf urllist', and
retry.

sh: yes/sources/authors/id/D/DC/DCONWAY/Text-Autoformat-1.04.tar.gz: No such fil
e or directory
Could not open >yes/build/DCONWAY000/Makefile.PL at (eval 26) line 4219

=============================

[fastforward]

I've never seen errors like that slinky :(
What version of Perl do you have and what platform?
You might have better luck installing them using the traditional method.

[slinky]

Not sure which version of perl but I went to CPAN to find the modules to install and not all of them are there. Here are the ones that are either not installed or that I couldn't find.

Bundle::mysql could be one of several different bundles.
Date::Parse isn't listed.
MIME::Decoder isn't listed
HTML::Entities isn't listed.

[fastforward]

Here's the links:

Bundle::mysql (http://www.cpan.org/authors/id/JWIED/Msql-Mysql-modules-1.2216.tar.gz) (part of the Msql-Mysql-modules bundle)
Date::Parse (http://www.cpan.org/authors/id/GBARR/TimeDate-1.10.tar.gz) (part of the TimeDate bundle)
MIME::Decoder (http://www.cpan.org/authors/id/ERYQ/MIME-tools-5.410.tar.gz) (part of the MIME-tools bundle
HTML::Entities (http://www.cpan.org/authors/id/GAAS/HTML-Parser-3.25.tar.gz) (part of the HTML-Parser bundle)

There's a good chance that these particular four are already installed on your system. They are very common.

[slinky]

You're a saint... :) I booted it up and realize that MIME::WordDecoder isn't installed. After searching all over the place I found out that it is part of the MIME-tools package, FYI... I'm going to try to install them now... :)

[fastforward]

I see you're using the vBPortal Slinky. It looks good! I must have a closer look at that when I find time. That's a nice domain name too! No doubt you have plenty of offers for it :)

[slinky]

Originally posted by fastforward
I see you're using the vBPortal Slinky. It looks good! I must have a closer look at that when I find time. That's a nice domain name too! No doubt you have plenty of offers for it :) Yes - the Portal is coming along very nicely. Testing for wajones and I've added in a few twists of my own! :) Now all I need to do is hire a designer just to design the overall site look. The domain name is a long story but it worked out really well for me... got it over 6 years ago, then was involved with the former mayor of New York, now I'm back in control... long story but great ride. :D

Almost there... some of the problems I've encountered could be that I'm using a virtual host and I've had to install the modules into another directory /usr/thelaw/perl/ which I created. The only error I'm getting now upon install and make are:

Warning: I could not locate your pod2man program. Please make sure, your pod2man program is in your PATH before you execute 'make'

Getting closer....

[carrillo]

Originally posted by fastforward
Here's the links:

Bundle::mysql (http://www.cpan.org/authors/id/JWIED/Msql-Mysql-modules-1.2216.tar.gz) (part of the Msql-Mysql-modules bundle)
Date::Parse (http://www.cpan.org/authors/id/GBARR/TimeDate-1.10.tar.gz) (part of the TimeDate bundle)
MIME::Decoder (http://www.cpan.org/authors/id/ERYQ/MIME-tools-5.410.tar.gz) (part of the MIME-tools bundle
HTML::Entities (http://www.cpan.org/authors/id/GAAS/HTML-Parser-3.25.tar.gz) (part of the HTML-Parser bundle)

There's a good chance that these particular four are already installed on your system. They are very common.


MMmm,
an scripts "that" check wich modules your need for this hack..!

no...yes...to much work :rolleyes:

carrillo

[mobilebbs]

Please Can Help Me ??

I found this error when i try use newnews.pl to import newsgroup into my forum but it so many error like this .. :D


DBD::mysql::db do failed: Duplicate entry 'from:' for key 2 [for statement ``SELECT wordid FROM word WHERE title = '??''']) at newnews.pl line 628.


Please Who can help me fix it ??? Thank you Very Much.. :D

[fastforward]

I've seen a few other posts with similar errors (not related to this hack). I believe the advice given was to empty the word and searchindex tables and reindex. you could also try just deleting that particular record 'where title = '¡»'.

If you don't want to either of the above, you can simply turn off usenet post indexing for one batch run using the usenet control panel.

[mobilebbs]

and Thank you very much.. :D I will do this ..

and 2. Question.

Sometime I try use newnews.pl to get some newgroups posts
but I have try many newsgroups , but why some group I just can not import any news into forum, But I am sure this newsgroup have many posts ??

would you can access newgroup "tw.bbc.comp.linux" or "tw.bbs.comp.hardware" ?? this two chinese newsgroups have many posts everyday . But I just can not import any things ??

[chrispadfield]

Well finally got this installed again and wow, looking brilliant
http://www.ascifi.com/forums2/forumdisplay.php?forumid=980

couple of things:

i) the merge into new thead for lost posts didn't work and gave this error;



<!-- Database error in vBulletin Control Panel: Invalid SQL: SELECT msgid,subject FROM usenet_article WHERE ORDER BY dtm ASC LIMIT 1
mysql error: You have an error in your SQL syntax near 'ORDER BY dtm ASC LIMIT 1' at line 1
mysql error number: 1064
Date: Thursday 31st of May 2001 05:37:13 AM
Script: /forums2/admin/usenet.php
Referer: http://www.ascifi.com/forums2/admin/usenet.php?action=manual&group=alt.tv.stargate-sg1&groupid=980
-->

i have a feeling this might be because i have an old version of mysql but amnot sure

2. Doing "new threads" instead gives this error:


Warning: Invalid argument supplied for foreach() in /home/ascifi/www/forums2/admin/usenet.php on line 399


but insert into thread worked like a charm.

3. News server

which one? I need:

i) cheapish, less than $15 per month
ii) no speed restriction, restriction done on monthly bandwidth
iii) does NOT post a footer. This is why i am leaving newsfeeds
iv) decent selection of newsgoups, ie need to be able to check first
v) and preferably uses a 3rd party credit card processor. I don't want to be easily liable for any spam muck ups that someone might attempt to do and get charged $150 an hour "clean up". They wouldn't be able to do this if they use paysystems/instabill which is one of the reasons i like newsfeeds.

Any suggestions?

[chrispadfield]

oh yeah, are you planning on finishing of a "vb2.0 final template set for newsgroups"? would be very cool ;)

[fastforward]

Chris,

The merge into lost threads and all other functions within that option need updating. I made a lot of table structure changes without changing this function. It was low on my priority list as I don't use it :) I willl get it fixed though.

As for a news feed, I switched to onlynews.com from newsfeeds. They are very fast and have no speed restriction, but the retention is pretty crap (4-5 days) and the number groups is quite low. I'm looking for another one myself. Let me know if you find one.



mobilebbs,

There are several reasons why posts may not get imported.
1) You haven't corrrectly mapped the local forum to the usenet group.
2) The article numbers have somehow got out of synch. (try resetting the numbers to zero for those groups)
3) you are pulling historical posts and have autopurge turned on and set too low. This will cause old posts to be immediately deleted after being imported.
4) The articles are all getting rejected for spam (unlikely)

[mobilebbs]

Originally posted by fastforward
mobilebbs,

There are several reasons why posts may not get imported.
1) You haven't corrrectly mapped the local forum to the usenet group.
2) The article numbers have somehow got out of synch. (try resetting the numbers to zero for those groups)
3) you are pulling historical posts and have autopurge turned on and set too low. This will cause old posts to be immediately deleted after being imported.
4) The articles are all getting rejected for spam (unlikely) [/B]

. Thanks .. Let me try .. Sorry My English is very bad . But Really
Thank for your replys.. :D

[v0n]

For some reason I get a little messed up html code with quoted replies in version 20010528.

Here's what i mean:
<br>
User <name@email.dom> wrote in message<br>
<a href="http://news:somerefnum@somenewserv.dom" target="_blank">news:somerefnum@somnewserv.dom</a>...<br>
<font size="1">&nbsp;&nbsp;<i><font color="blue<br>">>

Which obviously produces a nasty color far from blue.

I don't quite know how to locate the bug...

[fastforward]

Originally posted by v0n
For some reason I get a little messed up html code with quoted replies in version 20010528.

Here's what i mean:
<br>
User <name@email.dom> wrote in message<br>
<a href="http://news:somerefnum@somenewserv.dom" target="_blank">news:somerefnum@somnewserv.dom</a>...<br>
<font size="1">&nbsp;&nbsp;<i><font color="blue<br>">>

Which obviously produces a nasty color far from blue.

I don't quite know how to locate the bug...
There was a change in this section of code to allow for the size setting. However, I don't seem to have this problem on my setup. What other usenet display settings do you have set ie. indents, color etc. I have all set to yes on mine.

[v0n]

All set to yes, wrap 90, quote 1, West Europan-Latin1

[chrispadfield]

finding a news server is a nightmare, an incredibly disorganized and unprofessional looking sector incredibly annoying.

Anyway, realised that spamkiller.newsfeeds does not in fact add any signature so i am going to use that. Paul, why did you switch from newsfeeds? is there something i should know about?

i guess that for us retention does not matter as we are mirroring everything anyway apart from the initial pull of posts which will be less? is that correct?

finally, sorry to nag, are you going to do your template in v2.0 style. There were a lot of updates to the templates i think ):

[fastforward]

Originally posted by chrispadfield
Anyway, realised that spamkiller.newsfeeds does not in fact add any signature so i am going to use that. Paul, why did you switch from newsfeeds? is there something i should know about?

I left them out of principle after they accused me of spamming due to one post that did not have the body in the message, but did have the signature with my url! They didn't actually make the complaint obviously, but they still followed it up.

i guess that for us retention does not matter as we are mirroring everything anyway apart from the initial pull of posts which will be less? is that correct?

True, but I need a newsfeed for home aswell, with binaries and longer retention.

finally, sorry to nag, are you going to do your template in v2.0 style. There were a lot of updates to the templates i think ):
I'm afraid I won't be redoing them until I have finished redoing my britishexpats site. So it could be a month away.

[chrispadfield]

ok, the good thing about the spam is they can't just decide to charge us, one of the things i like about it.

i may work on a template set then,

cheers for the hack have been looking forward so long to get it installed! my mods are sick of me talking about it :)

[chrispadfield]

possible bug

i started a new thread in a usenet forum, while newnews.pl was running (while it was collecting threads). Anyway, the thread now sites in usenet_outgoing and won't get sent even when i run newnews.pl again.

[v0n]

I'm sorry for being such a pain but even after fiddling around with replacement variables i can't get rid of the
<font size="1"> <i><font color="blue<br>">>
bit.
For the time being I had to do upside down gimnastics and create two replacements:
{usenetquotecolor} is replaced by #031AAB">
and
}">> is replaced by >
just to make it into valid html (but with <br> is still at the end of every single line, which makes all quotes twice as long). I simply can't find where is that <br> coming from. Any ideas?

[NgtCrwlr]

I looked and couldn't find anything wrong, and I the same as fastforward don't seem to have the problem.

This is a long shot... check your vB replacement set
and make certain you don't have a carriage return after blue.

It will not show on the CP listing
{usenetquotecolor} is replaced by blue [edit] [remove]
you need to go into edit and press delete several times at the end of the word blue.

I hope this makes sense?

Cheers!

[v0n]

That was one of the first things i've checked. All the articles look ok in database:

[size=1]{usenetindent}[i][color={usenetquotecolor}]> (here some text)

so I guess whatever adds the <br> tags is between db and showthread.php.
I also have no custom replacement variables except for the usenetquotecolor ones....
:(

[zarkov]

fast forward,
I have spent some 4 hrs reading with interest all the posts today in this thread with amazement.

You realy unleashed a can of worms with this hack didnt you :)

I must praise you and the other members who have been with you on this from the start for your response times and dedication to resolving the problems.

Well done.

After reading all this i have a few questions i would like to aske before i install it on my m8s board to "Try".

1) Exactly which perl modules need to be installed for this script to run?
2) How can i find out which modules are already installed?
3) How do i find out what CRON is and if that is installed or usable or whatever it needs to be?

Sorry for the dumb Q's but im prety sure they havnt been asked/answered before and i know absolutly nothing about perl.

Thanks for the hard work and effort once again.

[fastforward]

Originally posted by Zarkov
1) Exactly which perl modules need to be installed for this script to run?

Bundle::DBI - DBI-1.16.tar.gz (http://www.cpan.org/authors/id/TIMB/DBI-1.16.tar.gz) (should already be installed)
Bundle::DBD::mysql - part of the Msql-Mysql-modules package (http://www.cpan.org/authors/id/JWIED/Msql-Mysql-modules-1.2216.tar.gz) (should already be installed)
HTML::Entities - part of the HTML-Parser package (http://www.cpan.org/authors/id/GAAS/HTML-Parser-3.25.tar.gz) (should already be installed)
MailTools - MailTools-1.15.tar.gz (http://www.cpan.org/authors/id/GBARR/MailTools-1.15.tar.gz) (should already be installed)
Net::NNTP & Net::SMTP - both part of the libnet package (http://www.cpan.org/authors/id/GBARR/libnet-1.0703.tar.gz) (probably already installed but older versions will not have the NNTP module)
MIME::WordDecoder - part of the MIME-tools package (http://www.cpan.org/authors/id/ERYQ/MIME-tools-5.410.tar.gz) (MIME-tools is probably already installed but older versions do not have the WordDecoder module)
Date::Parse - part of the TimeDate package (http://www.cpan.org/authors/id/GBARR/TimeDate-1.10.tar.gz) (maybe installed)
Text-Autoformat - Text-Autoformat-1.04.tar.gz (http://www.cpan.org/authors/id/DCONWAY/Text-Autoformat-1.04.tar.gz) (probably not installed)
Email-Find - Email-Find-0.03.tar.gz (http://www.cpan.org/authors/id/MSCHWERN/Email-Find-0.03.tar.gz) (probably not installed)
Email::Valid - Email-Valid-0.13.tar.gz (http://www.cpan.org/authors/id/MAURICE/Email-Valid-0.13.tar.gz) (probably not installed)
URI::Find - URI-Find-0.04.tar.gz (http://www.cpan.org/authors/id/MSCHWERN/URI-Find-0.04.tar.gz) (probably not installed)

2) How can i find out which modules are already installed?

The easiest way is to run the script. It will tell you if it can't find a module.

3) How do i find out what CRON is and if that is installed or usable or whatever it needs to be?

Cron is installed along with operating system in every unix and linux flavour I have come across. Type: 'crontab -l' at the command prompt. As long as it doesn't give you a permission denied error, you are ok.

[NgtCrwlr]

Originally posted by Zarkov
2) How can i find out which modules are already installed?

I find the following script handy for the listing installed perl modules. I posted this a while back but for some reason the page that it is on is generating a vB parse error. So here it is again.

Download:
http://www.makeitsimple.com/download.php?op=geninfo&did=6

Working example:
http://www.makeitsimple.com/cgi-bin/modules.cgi

I just do a find(Ctrl+F) on the page for whatever module is in question.

Cheers!

[zarkov]

wow thanks for the quick responses guys much appreciated

[slinky]

OK... I'm moving along with the usenet gateway and getting closer. (For those of you who are new to perl and hacking, this is not a hack for the light of heart!) The problem I had installing the perl modules is that I am hosted by a virtual host who does not technically support perl modules, some of which require superuser access. However, I was able to get around it a bit by installing the modules and perl in a directory off of my root directory. Typing in usr/bin/perl or whatever would not work because I don't have control to the absolute root of the server, only mine. Hope this helps and I'm back at it...

[tamarian]

Originally posted by slinky
(For those of you who are new to perl and hacking, this is not a hack for the light of heart!) [/B]

Ditto!

Be afraid, be very afraid! :)

No, seriously, you need your own test environment, with PHP and Perl to try things out first, and have backups.

This in no way reflects in Fastforwards skills, in my book this IS the best and most elegant vb hack I ever installed. The reason you should be careful is that this hack requires significant changes in you database since it essentially adds a functionality VB never was designed to do. And it does it very well. It doesn't add a little thing here and there like most hacks, it adds a LOT.

So, if you know your stuff, go ahead. If you're not sure, ask first.

[zarkov]

I couldnt agree more which is why i asked ;)

But by the souds of it id be best advised not to install and test on a working board with NO perl knowledge/ experience then ill leave alone and hope that maybe something similar will be written in PHP ( if its possible, but i thought that php could do everything that perl can do and more)

thanks for the forewarnings and all the advice though people it IS really apreciated

[tamarian]

I'd be careful even if it was written in PHP! :)

[fastforward]

Originally posted by Zarkov
but i thought that php could do everything that perl can do and more
You've got that the wrong way round. Having said that, this particular app could be written in PHP but you wouldn't have all the modules to help you out. Plus the fact that this is executed in the background using cron. Not many hosts have the PHP executable installed that allows you to run php scripts standalone. In fact many people don't even know that you can run php scripts as standalone programs.

[NgtCrwlr]

Time to get serious about a usenet news feed. Up until now I've been all smiles, getting the gateway setup has been a blast.

Now however, reality hits and hits hard. I've had my head so buried in getting the gateway installed that I hadn't given much thought to tuning it on for mainstream use. Until now I've had all my usenet forums locked to read only, then I started thinking about the spam issue. So I stared reading TOS agreements at a bunch of the usual less than $25/mo services. Oh-oh time... I know I read a few posts here of people wondering about the same issue and at the time I just blew it off thinking ah heck, I'll worry about that when I get there.

Well I'm there/here and I'm worried. Not only is it against the TOS of just about any service to pull/suck news... most of them specifically state they are for individual use only. Not to mention the up to $200/hr spam cleanup charges. Several renegade BBS users could bring a world of hurt on the web master. Especially when we would be breaking the TOS even before getting caught for spam problems. Not a good picture.

How about a few of us group together and pre-pay for one *real* feed that we could all use? This would be cheaper than doing it alone.

Here's a few snips of what I found so far...

Usenet NewsServer Club
Account Usage - UNSC accounts are single-user accounts. This means that only one login from your account is allowed at a time, and the owner of the account (or their immediate family) is the only person allowed to use that account.
http://www.alibis.com/

NewsGroups.com
I will not sell, redistribute, or copy in any way the contents of this site. I will not allow anyone to use my password or post SPAM of any type. If I do post, I do so at my own risk.
http://www.newsgroups.com/

1usenet.com
INTENDED USE - Accounts are intended for personal use only, commercial use is prohibited without prior written authorization

We have ZERO tolerance for Spammers or Flooders. Posting is a privilege, not a right, if you post SPAM and or flood any group, your account or posting privileges may be revoked WITHOUT PRIOR WARNING, and you may be billed (at the discretion of the System Administrator) for the cleanup, at $200.00 (US) per hour, plus any Collections and or Attorney's fee's incurred in collection of this fee. 30 day auto billing for 1 year
http://www.1usenet.com/

Usenet-X.com = popup crap - Don't even bother
SPAM AND FLOODING - We have ZERO tolerance for Spammers or Flooders. Posting is a privilege, not a right, if you post SPAM and or flood any group, your account or posting privileges may be revoked WITHOUT PRIOR WARNING, and you may be billed (at the discretion of the System Administrator) for the cleanup, at $200.00 (US) per hour, plus any Collections and or Attorney's fee's incurred in collection of this fee.
Posting Footer - usenet-x.com reservers the right to attach an advertisement to the bottom any posts you make through our service. If you object to this you have the option of posting through spamkiller.usenet-x.com, which does not attach the footer.
INTENDED USE - Accounts are intended for personal use only, commercial use is prohibited without prior written authorization from usenet-x.com.

[NgtCrwlr]

Here's a couple possible candidates.

Ball and Ball, L.P
$59.95/month Pull Newsfeed - includes accepting a return feed of your local posts. You are expected to enforce a strong anti-spam policy. (limited to 250 Megabytes of downloads per day)

$99.95/month Pull Newsfeed - includes accepting a return feed of your local posts. You are expected to enforce a strong anti-spam policy. (limited to 500 Megabytes of downloads per day)

$149.95 /month Pull Newsfeed - includes accepting a return feed of your local posts. You are expected to enforce a strong anti-spam policy. (limited to 1000 Megabytes of downloads per day)

Usenet Acceptable use Policy: We have a strong anti-spam policy. If you spam Usenet through our servers, we will terminate your account and charge you for any costs we incur in dealing with complaints and cleaning up your mess. Posting of illegal content is NOT allowed on our servers and is grounds for account termination without a refund. Groups carried are subject to change without notice.

For our Usenet service, we use the Dnews server software. Our service is aimed at the user who wishes to participate in Usenet discussion groups. If it's porn, warez, or mp3 files you're looking for, we're not the choice for you. If it's a fast server with decent retention and good spam filtering, we're the choice for you. We try to make our service family friendly.
http://www.bnb-lp.com/Usenet.html

Eskimo.com Usenet Services
For Organizations - We provide NNTP read/post access to other sites, ISPs, businesses, BBSs, and other organizations, for $600/year, or $180/quarter, or $70/month per domain. If you're looking for a way to provide Usenet News services to your customers without the expense of maintaining your own news server and you are looking for a solution that doesn't impose a per subscriber or per megabyte charge consider using our news service as an option.


NOT a candidate just more of what we don't want!

NewsGuy.com
Newsguy are intended for the use of its subscribers and is not intended for re-distribution, or as an unauthorized news feed to another service. Downloading article quantities indicative of pulling a newsfeed from either service is unacceptable. Subscribers connecting to the service with news retrieval software which is designed to copy the contents of large quantities of newsgroups, or sustain multiple news connections, will have their service terminated. *snipped*

The cost for a commercial newsfeed is $400 per month. The Net.* hierarchy of newsgroups requires extra setup at the customer's site to insure compliance with their "sound site" policy. Recommended minimum equipment to fully utilize our newsfeeds:
A Pentium server, 400 Mhz or faster, 500 Mbs of RAM, 64 Gbytes of disk storage, INN news server software. A bused pair of T1 with combined 3 mbps available bandwidth.
Newsguy Newsfeeds cannot be propagated outbound to servers or resold. Only newsfeeds back to Newsguy's servers are allowed. As Usenet news is always evolving, Newsguy cannot guarantee the availability of any group(s), and we strongly urge you to review our service policy prior to signing up for our newsfeed.

[fastforward]

What I've done is installed a news server on my machine that pulls from all the other servers I'm interested in and then I just connect to my local server to get the news. You will never be able trap 100% of all spam, but with the hack spam filters and the DNews filters you should be quite safe.

As for the 'no pulling' clause. You will be pulling an insignificant amount of news in comparison to anyone who happens to download a few mp3s or a couple of binary pics each day. You won't even register on their meters.

The commercial clause is debatable. Unless you are charging subscriptions for access to your usenet forums you could argue your case.

But the bottom line is; the amount of news you are actually downloading really is tiny. As long as you stay away from the binary groups I wouldn't worry too much.

[Gilby]

NgtCrwlr, how many newsgroups do you plan on pulling? If it's just a few, then you should have no problem with most of the cheaper ones that are meant for individual use.

[NgtCrwlr]

Right now I have only 9 groups and don't plan to have more than 25. I know this is tiny and the bandwidth of even a history pull would not register on a meter. It still doesn't make me comfortable.

What about when/if any of us get popular and you end up with 500 to 2000 posts or more per day going out to usenet? Hmm one password and 1000K outgoing posts each day, I think that one might trip a red flag or two.

I think the biggest reason for my fear is that I'm not on a dedicated box yet and can't do some of the cool things that you dedicated guys can. If I had a usenet feed service send a spam alert to my current host I could effectively have my sites totally shut down, not just the news feed. Even if the problem was with some trouble maker on the BBS it still filters down to me.

So I guess it comes down to a few choices for me either leave the usenet forums read only and use free servers, pay the money for either a dedicated server, or pay the higher buck news service that understands and accepts what we are doing.

I'm still searching, who knows I might come up with a service that will welcome us all with open arms. :D

[fastforward]

That Ball and Ball account seems very cheap. I was paying $30 for my newsfeeds account before I left them, and that was just a normal personal account for home use. I might take a closer look at them myself :)

Keep us updated if you come across anymore bargains.