[RELEASE] Usenet Gateway for vB 2.01

[fastforward]

Originally posted by Brian Cruz
I've also noticed that when outgoing posts which contain quotes appear on usenet, the quoted text has blank lines between them. They look like this:

Juan F. Lara wrote:
> Years ago I heard that Warner Brothers Television Animation was going to

> make an "Osmosis Jones" TV series to go along with the movie. Whatever

> happened to that TV series? Are they still going to make it?


Is that normal for the script or did I set up something wrong?
No, this is not normal. It looks like a nasty bug. Why didn't anyone tell me about this before? :eek:

I'll fix it this weekend. In the meantime, if you go to line number 1029 in newnews.pl and change the existing line that currently reads:

push(@article," > $q\n\n");

to

push(@article," > $q\n");

[Brian Cruz]

Originally posted by fastforward

No, this is not normal. It looks like a nasty bug. Why didn't anyone tell me about this before? :eek:

Thanks, I've made the change. Another thing I noticed is that when I quote a usenet post which contains a link, it appears in my new reply window as this (I added the spaces between the brackets so it shows up here):

[ q1 ]: [ url="http://130.94.172.139/forums/" ]http://130.94.172.139/forums/[ /url ][ /q1 ]

And on usenet it shows up like this:

> http://130.94.172.139/forums/http://130.94.172.139/forums/

Is that also a bug?

[Pie'oh'pah]

Originally posted by fastforward

Yeah, that should be easy. Usenet posts made by forum members can be identified by a '-1' in the isusenetpost column in the post table.

It might take me a few weeks to get around to it though. I'm in the middle of re-doing BritishExpats.com. It's still running vB 1.1.6 so I think it's about time I did something about it. :p

You can make the change yourself if you like. It's easy peasy. Find the line that was added by the usenet hack in search.php. It should read:
$wheresql.=" AND thread.isusenetpost=0";
Change that to:
$wheresql.=" AND thread.isusenetpost=0 AND post.isusenetpost <= -1";

If you just want threads that were started by members, change it to:
$wheresql.=" AND thread.isusenetpost <= -1";


bitbender:
Thanks for your kind words. :D


unfortunatly im getting this when doin those additions:

Database error in vBulletin: Invalid SQL: SELECT COUNT(*) AS threads FROM thread

WHERE thread.visible=1 AND 1=1 AND thread.lastpost>=998680523 AND thread.open<>10 AND thread.isusenetpost=0 AND post.isusenetpost <= -1
mysql error:
mysql error number:
Date: Friday 24th of August 2001 01:51:05 PM
Script: /~unreal-a/vbulletin/search.php3?s=&action=showresults&getnew=true&searchid=551
Referer:

Any hints? Thanx in advance!

-Alex

[fastforward]

Originally posted by Pie'oh'pah



unfortunatly im getting this when doin those additions:

Database error in vBulletin: Invalid SQL: SELECT COUNT(*) AS threads FROM thread

WHERE thread.visible=1 AND 1=1 AND thread.lastpost>=998680523 AND thread.open<>10 AND thread.isusenetpost=0 AND post.isusenetpost <= -1
mysql error:
mysql error number:
Date: Friday 24th of August 2001 01:51:05 PM
Script: /~unreal-a/vbulletin/search.php3?s=&action=showresults&getnew=true&searchid=551
Referer:

Any hints? Thanx in advance!

-Alex
Looks like it's not as easy as I thought. the post table is not in the from clause. You'll have to wait until I can think of another way.

[Pie'oh'pah]

Sure No Problem

just add this to your to do list, if you like :)

Thnx for fast response!

-Alex

[JCDR]

Originally posted by fastforward

It goes out the next time you run the script. How long it takes before it shows up on the news server is dependent on each server. Assuming the post was accepted, and you are reading from the same server to which you posted, it should be only a few minutes.

Sorry FF I didn't explain fully, the message was posted but the subject of the message was not so all that was posted to the newsgroup was a blank message but the subject was there.

[webhost]

Fastforward you might wnat to take a look at this thread then come back and read what happens when you do this. This will happen to all users with next version if not corrected.

http://www.vbulletin.com/forum/showthread.php?s=&threadid=26126


I did what the above thread mentioned and it did work after I replaced script in functions.php. Then I updated counters in control panel the forums that were empty it cleared out the last username and date/time of the last post.
The problem came with the usenet groups even though they all had tons of post's in them the last post column on all of them also got cleared out. The other regular forums that did have post's in them did not. So if even though this does fix a bug with vb it screws with the usenet hack a little. I ran the pl file again and any newsgroups that had new posts they were installed in the last post column so this is just a small problem. Just wanted everyone that is using the usenet to know that until this is somehow fixed everytime you update counters if you make the change mentioned in the above thread that this will happen.

Joey

[DarkReaper]

A small bug:

In new reply, at the very top I get "8/10/08" before any of the header loads. I've figure out that when there's stuff way up there, its usually because a variable is being called that has no value. I find this strange though, as the only change I have made to newreply.php is the single code change for the usenet hack, and the template for it is still the same.

Any ideas?

[Dibble]

Hi,

I have installed the hack on my server
v20010820 (for 2.03)

I think this is a great hack, but I'm having trouble :(

========================
When I run newnews.pl
- I succesfully connect to the news server
- I get a load of NNTPERROR: 430 No such Article
- Then says fetching xxxx... OK for a while
- NNTPERROR: 421 No next article
- NNTPERROR: 423 No such article in group
- NNTPERROR: 430 No such article a few times
- -> Processing article batch -> requested 56 messages 0 not available or rejected
- -> inserting new threads from alt.comp.periphs.cdr.mitsumi

Then :(
Query failed
INSERT INTO post (blah blah blah)

DBD::mysql::db do failed: You have an error in your SQL syntax near '998959523,994399103,'This message posted with trial version of Express News Post' at line 1 at newnews.pl line 741, <SOCK1> line 2375
=====================

Anyone shed any light on this?

Any help greatly appreciated

Cheers

Dibble

[fastforward]

Originally posted by Dibble
Hi,

I have installed the hack on my server
v20010820 (for 2.03)

I think this is a great hack, but I'm having trouble :(

========================
When I run newnews.pl
- I succesfully connect to the news server
- I get a load of NNTPERROR: 430 No such Article
- Then says fetching xxxx... OK for a while
- NNTPERROR: 421 No next article
- NNTPERROR: 423 No such article in group
- NNTPERROR: 430 No such article a few times
- -> Processing article batch -> requested 56 messages 0 not available or rejected
- -> inserting new threads from alt.comp.periphs.cdr.mitsumi

Then :(
Query failed
INSERT INTO post (blah blah blah)

DBD::mysql::db do failed: You have an error in your SQL syntax near '998959523,994399103,'This message posted with trial version of Express News Post' at line 1 at newnews.pl line 741, <SOCK1> line 2375
=====================

Anyone shed any light on this?

Any help greatly appreciated

Cheers

Dibble
The 'No such Article' messages are normal. Read back and you'll see explanations on this.

I'll need to see the full SQL error to tell you what's wrong with the other thing.

[Dibble]

Thanks for the quick reply

I'll post the full error when I have a sec. (I use SSH and I cant paste out of the damn thing....)

Anyway its 03:15 in the UK, and I shoudl probably turn up for work later.

Cheers

Dibble.

[webhost]

try blocking everything with the mouse and using ctrl key then go to notepad place your cursor where you want it and it paste.

[Dibble]

:)
I defeated the evil SSH and used Telnet (no-one looking at my password? Good.)

So, here is most of the error message.

=============
There was stuff before this............
NNTPERROR: 430 No Such Article
NNTPERROR: 430 No Such Article
NNTPERROR: 430 No Such Article
NNTPERROR: 430 No Such Article
NNTPERROR: 430 No Such Article
-> processing article batch...
-> requested 56 messages... 0 not available or rejected.
-> inserting new threads from alt.comp.periphs.cdr.mitsumi

Query failed:
INSERT INTO post (title,allowsmilie,threadid,dateline,nntpdateline, pagetext,visi
ble,msgid,ord,username,userid,nntpfrom,ipaddress,i susenetpost) VALUES ('What do
you think? I want to know!!!',0,,998965144,994399103,'This message posted with t
rial version of Express News Poster\n__________________________________________ _
__________________\n\nWant to voice your opinion or belief? Visit my new site Ca
lled "Alternatives To Popular Beliefs"\n(www.jrbjr.freeservers.com (http://www.jrbjr.freeservers.com/
)) I have a chatroom, messageboard and poll. I a
m looking for new topics,\nso please send me ideas for topics you would like to
see on my site.','1','<3sc17.560287$ho6.36488278@news5.aus1.giganews.com>','0','
Jrbjr',0,'&quot;jrbjr&quot; &lt;jrbjr@digitalexp.com&gt;',0,1)

DBD::mysql::db do failed: You have an error in your SQL syntax near '998965144,9
94399103,'This message posted with trial version of Express News Post' at line 1
at newnews.pl line 741, <SOCK1> line 2409.
============

Thanks for any advice,

I've forked out for a paid usenet feed so it'll be cool if I can get this hack to fit around it as well. Its a darn site better than the usenet feed I get with my paid ISP.


Cheers

Dibble

[fastforward]

Originally posted by Dibble
So, here is most of the error message.
Query failed:
INSERT INTO post (title,allowsmilie,threadid,dateline,nntpdateline, pagetext,visi
ble,msgid,ord,username,userid,nntpfrom,ipaddress,i susenetpost) VALUES ('What do you think? I want to know!!!',0,,998965144,994399103,'This ... etc, etc...

As I thought. The threadid (third value in the insert clause) is null. You need to upgrade your Perl DBI and MySQL modules.

[Dibble]

Argh.

Sounds serious.

Please post rough details, as I will have to ask my hosting company to do this for me.
(Unless it is very easy)

Cheers

Dibble.

[DarkReaper]

Originally posted by DarkReaper
A small bug:

In new reply, at the very top I get "8/10/08" before any of the header loads. I've figure out that when there's stuff way up there, its usually because a variable is being called that has no value. I find this strange though, as the only change I have made to newreply.php is the single code change for the usenet hack, and the template for it is still the same.

Any ideas?

...

Need help :)

[fastforward]

Originally posted by DarkReaper


...

Need help :)
No idea. If I can reproduce it I'll let you know.

[JCDR]

Originally posted by JCDR
Sorry FF I didn't explain fully, the message was posted but the subject of the message was not so all that was posted to the newsgroup was a blank message but the subject was there.

Anyone any idea why the message gets posted to the newsgroup but there is no 'message' just the heading??

[Dibble]

I can import news ok now, but I cant seem to export locally sent messages to usenet.

============
bash$ perl newnews.pl


Connecting to news-central.giganews.com:119... Connected
Sending authentication info... Authenticated and logged in

Pulling alt.comp.periphs.cdr.mitsumi... No new messages.
Clean disconnection from news-central.giganews.com:119

Expiring usenet articles older than 150 days...
-> alt.comp.periphs.cdr.mitsumi... OK
Expire completed.
=================

This doesnt seem to be posting any back to usenet.
Anyone have any pointers about where I should look to fix this?

FYI - my board is at www.mitsumicdr.com/forums

Cheers,

Dibble.

[DarkReaper]

Originally posted by fastforward

No idea. If I can reproduce it I'll let you know.

ROFL. Flame me, kick me, torture me, I deserve it. Before the <?php... in newthread.php there was...guess what! 8/10/08...how the hell did that get there and what is it? lol!

[fastforward]

Originally posted by DarkReaper


ROFL. Flame me, kick me, torture me, I deserve it. Before the <?php... in newthread.php there was...guess what! 8/10/08...how the hell did that get there and what is it? lol!
:p

[webhost]

Question when reindexing search index on board via the control panel under update counters, does the usenet groups always get included or is there away to stop this from happening?

[fastforward]

Originally posted by webhost
Question when reindexing search index on board via the control panel under update counters, does the usenet groups always get included or is there away to stop this from happening?
Once the usenet posts are in your forum they are treated just like any other post.

[webhost]

Question, when running the script it fetchs messages rather quickly, but it boggs down some when it starts finding replies. Is there any way to speed this up in the script?

[george_proost]

Hi FF and other hack users,

Hmm... special case me thinks..


I use multiple newsfeeds
The newsgroup in question has it's own footer message
the posting user uses show signature
signature and return email usenet data is present
spam filtering is not used outgoing


symptoms:
the correct signature and footer message are shown in usenet newsgroup but the post text 'body' is missing. Any ideas.

[Joop]

I have had this great hack working since spring.
I upgraded to the latest version at last week and everything seems to be ok, but I cannot find new threads or replies from news server.
This started last friday, when one user send me a mail and told me, that new threads doesn't show on news servers. Today I noticed, that even the replies dissapears somewhere.

When testing, console says:
Connecting to news.kolumbus.fi... Connected
Posting message by mphelsinki to sfnet.test... Sent

Pulling sfnet.test... No new messages.
Clean disconnection from news.kolumbus.fi

What could be wrong ?

-Joop

[shadowbreed]

I don't know if this has been covered before but i have the following problem:

When using new thread, my threads show up fine on the newsserver, but when using reply or quote, nothing gets send

newnews.pl doesn't give any errors

any ideas?

[bitbender]

I have some questions:


The expiration routines. How do they work? Under what criteria do they expire threads ?
Should usenet posts be threaded into forums of the same name for easy segmentation? (like COMP.OS.LINUX.SETUP)
If I start a thread in one of the segregated usenet groups, will it thread it out to the named usenet group for reply ??


Thanks!

[AJR]

I just installed the usenet hack... had to wait 2 days for the host to install all the modules, but everything seems to be in order now! I just newnews.pl for the first time... the message gathering went smoothly. At the end of fetching the first 500 messages, it started to process the batch. From here, this is what is shown in telnet:

-> processing article batch...
-> requested 500 messages... 0 not available or rejected
-> inserting new threads from rec.games.pinball
-> inserted 166 threads
-> finding replies...

Query failed:
TRUNCATE TABLE usenet_article_loader

DBD::mysql::db do failed: You have an error in your SQL syntax near 'TABLE usenet_article_loader' at line 1 at newnews.pl line 741, <SOCK1> chunk 21940.


Any ideas? Is this bad? All the tables have been created. I can see the 166 threads it inserted, but none have replies. Before I grab the next batch of 500, I'd like to know what to do. The site is at http://www.vpforums.com but I have the Usenet groups hidden (shown only to admins) while I'm setting them up. Any help would be appreciated.

[AJR]

Can anyone help with my problem above please...

[AJR]

The problem I have above is with the 'TRUNCATE' MySQL command. If I try and run the command 'TRUNCATE TABLE usenet_article_loader' it comes back with a MySQL error. Is this a valid command for MySQL? If so, what version of MySQL supports it? I've been to the MySQL manual website and from what I understand, TRUNCATE basically drops and re-creates a table wiping out all data in that table but keeping the table structure. It's supposed to be more efficient than deleting each record individually.

My question is since the 'TRUNCATE TABLE usenet_article_loader' doesn't work on my server(s) that I've tested it on, will the 'DELETE FROM usenet_article_loader' do the same thing? Can I change the queries in the newnews.pl file from TRUNCATE TABLE' to 'DELETE FROM' ??

Right now what I'm doing is I remmed out the 'TRUNCATE TABLE' command from the newnews.pl file so it doesn't error out when running from the telnet command. After each group of messages is received (500 at a time), I'm going into phpMyAdmin and Emptying the usenet_article_loader table, then running the next batch of 500.

Also one other question... when the newnews.pl file gets run, I have set it to pull articles with less than 100 lines for right now. How long should it be taking to pull 500 messages? It seems REALLY slow (15 seconds/article). I use mammothnews.net and in my Agent Newsreader, it pulls 1500 article headers in about 20 seconds! So I know it's not the mammoth server. It did run a lot faster on Friday when I first got the TRUNCATE error (2 seconds/article). I have 11,000 articles to pull and it will take a really long time to get my news up to date!

Any help is appreciated...I would like to get this hack working completely and running on my site.

[fastforward]

You can replace TRUNCATE with DELETE. There won't be any problems (except slight speed impact).

As for the speed of articles being pulled; you should be getting between 1 and 3 articles per second. If you are sure it's not your news server, then it's probably something to do with the combinations of version of Perl and the modules. Not much you can do if you don't have full control over which versions you can install.

[KimG]

Hi,
I have just installed this great hack; after waiting many months for my host to switch me to my own server, and then install the modules!

Seems to work fine. I just have a couple of questions:

1) Spam control. Is it possible to only retrieve messages if they contain a certain word? How would I do this?

2) I have noticed that dbforums and other sites running this hack have the section of the vbulletin home page layout differently to other sections - ie. no moderators and differnt styles. How do you do this. Please could you just point me in the right direction!

Great hack - you should have charged :)

Kim

[AJR]

KimG... I'm working on changing the layout for my usenet group forums to be different than the normal forums. You have to add a new Style and Template set (I called them 'usenet'). Then change the style to use the 'usenet' templates. Then modify the forums to use the 'usenet' style. Hope this helps!

[KimG]

Originally posted by AJR
KimG... I'm working on changing the layout for my usenet group forums to be different than the normal forums. You have to add a new Style and Template set (I called them 'usenet'). Then change the style to use the 'usenet' templates. Then modify the forums to use the 'usenet' style. Hope this helps!

Thanks - would that alter the way that the newsgroups are displayed on the main forums page though. I just want a section at the bottom of my forums homepage (index.php) which list the newsgroups, maybe without the moderator columns etc.

[bitbender]

Originally posted by bitbender
I have some questions:


The expiration routines. How do they work? Under what criteria do they expire threads ?
Should usenet posts be threaded into forums of the same name for easy segmentation? (like COMP.OS.LINUX.SETUP)
If I start a thread in one of the segregated usenet groups, will it thread it out to the named usenet group for reply ??


Thanks!


???

[fastforward]

Originally posted by bitbender

The expiration routines. How do they work? Under what criteria do they expire threads ?

Should usenet posts be threaded into forums of the same name for easy segmentation? (like COMP.OS.LINUX.SETUP)

If I start a thread in one of the segregated usenet groups, will it thread it out to the named usenet group for reply ??

The expire is done on 'number of days old', configurable via control panel.

You don't have to name the forum the same as the newsgroup. The mapping is done via the control panel. You can assign any forum to contain usenet posts.

[fastforward]

Originally posted by KimG
1) Spam control. Is it possible to only retrieve messages if they contain a certain word? How would I do this?

No, not possible and there are no plans to add it.

[fastforward]

Originally posted by fastforward

No, not possible and there are no plans to add it.
Actually, you can do it. Just create a regular expression that rejects everything except the ones you want. You'll need to read up on regular expressions to get it right.

[Lee Herron]

Is there a simple test to confirm if you have nntp installed on your box or not? Would prefer to know before I attempt to install one.

Man, a thread search would be nice on a thread this long uh?

[fastforward]

Originally posted by Lee Herron
Is there a simple test to confirm if you have nntp installed on your box or not? Would prefer to know before I attempt to install one.
If you mean the Perl NNTP modules; the best test is to run the script and see what fails :) It will tell you which modules it can't find.

Man, a thread search would be nice on a thread this long uh?
You should've seen the last thread! :p

[bitbender]

I had a weird error situation requiring hours of recovery(thx VB Staff for the how to's!!:) ) for some non-usenet threads that disappeared. I'm pretty sure that it was not related to this addition, but wanted to know for sure. Your answer clarified it well.

thanks!

[psx-dude]

So this will allow me to mirror a newsgroup on my server? Won't that take up a LOT of database/disk space?

[AJR]

Well this is strange... I am testing new threads and replies in my usenet group. When the message is sent, I open up my Agent newsreader and I see the messages that were sent from the forum, BUT the first lines are missing. For example, I type in:

Testing the new usenet group reader from http://www.vpforums.com

Come play actual pinball machines on your PC!

In Agent, the message shows up only as:
Come play actual pinball machines on your PC!

I sent a reply to another usenet message as well that was only 1 line, and it showed up in the correct message thread in Agent, but only showed my signature. The text I typed was missing! So I'm assuming it's chopping off my first line or 2 of text. Any reason for this? I have the SPAM filter on outgoing messages set to OFF. Everything else looks good! HELP!!

[KimG]

Originally posted by fastforward

Actually, you can do it. Just create a regular expression that rejects everything except the ones you want. You'll need to read up on regular expressions to get it right.

Hi,
I've been reading up and have tried to construct an expression:

>"xxx[a-z]"

where xxx is the word that you want to retrieve followed by any letter in the header. Any other posts would be filtered out as spam? Is this along the right lines??

Thanks!

[hostbet]

Can you tell me what I'm doing wrong?

This is the error:

Database error in vBulletin: Invalid SQL: SELECT iconid, title FROM thread WHERE forumid= ORDER BY lastpost DESC LIMIT 1
mysql error: You have an error in your SQL syntax near 'ORDER BY lastpost DESC LIMIT 1' at line 1
mysql error number: 1064
Date: Friday 21st of September 2001 07:43:38 AM
Script: /forums/editpost.php
Referer: http://helpdesk.hostbet.com/forums/editpost.php?s=&action=editpost&postid=5

[shri]

Fastforward,

I've posted some perl code here .. which may help decode binary attachments.

http://www.vbulletin.com/forum/showthread.php?threadid=28992

If its not a priority for you ... Hopefully someone here is a perl guru who can help out?

I am trying to get clari.* content into the board and they have a lot of articles which come attached with gif / jpg photos.

[DarkReaper]

Who's posted doesn't work with this(2.0.3). When I click on the reply count it says "{Name of thread starter}: {number of replies}" and that's it.

[shadowbreed]

Originally posted by shadowbreed
I don't know if this has been covered before but i have the following problem:

When using new thread, my threads show up fine on the newsserver, but when using reply or quote, nothing gets send

newnews.pl doesn't give any errors

any ideas?

Still have this problem, can anyone offer some help?

Now just MY users get to see the replies and not the people on the newsserver :(

Now i also have the following problem:


-> fetching 39349... OK -> fetching 39350... OK -> fetching 39351... OK -> fetching 39352... OK -> fetching 39353... OK -> fetching 39354... OK -> fetching 39355... OK NNTPERROR: 421 No Next Article NNTPERROR: 423 No Such Article In Group NNTPERROR: 421 No Next Article NNTPERROR: 423 No Such Article In Group NNTPERROR: 421 No Next Article -> processing article batch... -> requested 303 messages... 0 not available or rejected. -> inserting new threads from alt.macromedia.flash -> inserted 0 threads -> finding replies...

As you can see it does process everything but nothing gets inserted???

Can anyone please help??

[shadowbreed]

Has anyone had these problems??

[mkilty]

On my outgoing posts they do not have the text that the user typed in there. I gave it a test and the same thing happen except one line at the end of my message made it through.

Example:
------------------
The yellow brown fox jump over the fence. The yellow brown fox jump over the fence. The yellow brown fox jump over the fence.

Nice day....
----------------

The only text that made it through was "Nice day..." "SPAM filter" as well as "Apply replacement rules to outgoing posts" is set to OFF.

Help????

Michael



Originally posted by AJR
Well this is strange... I am testing new threads and replies in my usenet group. When the message is sent, I open up my Agent newsreader and I see the messages that were sent from the forum, BUT the first lines are missing. For example, I type in:



In Agent, the message shows up only as:


I sent a reply to another usenet message as well that was only 1 line, and it showed up in the correct message thread in Agent, but only showed my signature. The text I typed was missing! So I'm assuming it's chopping off my first line or 2 of text. Any reason for this? I have the SPAM filter on outgoing messages set to OFF. Everything else looks good! HELP!!

[fastforward]

I'll look into it, but it's not happening with me.

[YourHostSucks]

Ok it sems to not feel like working...

I get this


Connecting to news.*****.***... Connected

Pulling alt.php... No new messages.


It is not getting anything...... ;(

Any Ideas?

[hacker]

[2) I have noticed that dbforums and other sites running this hack have the section of the vbulletin home page layout differently to other sections - ie. no moderators and differnt styles. How do you do this. Please could you just point me in the right direction!

Great hack - you should have charged :)

Kim

Yes great hack indeed. Does anyone have a template to share? I love the way it's laid out but need help in this area.

[hacker]

UP

[shadowbreed]

fastforward could you please help me with my problem above??

[YourHostSucks]

Originally posted by YourHostSucks
Ok it sems to not feel like working...

I get this



It is not getting anything...... ;(

Any Ideas?

? :D

[fastforward]

Originally posted by shadowbreed
fastforward could you please help me with my problem above??
If you mean the empty posts, I don't see the error, so I can't really help. I've attached my newnews.pl. Compare it with yours and make sure it's the same. Pay particular attention to the post_outgoing_t, post_outgoing_p and post_outgoing functions.

[Shaman]

I'm having the problem that posts made from my board often have no text in the message when viewed on the newsgroup server - except the usenet signature and the tagline from the software.

However, if you type in quite a bit of data (say, 400 or more characters), it will post the message - in my tests, at least.

I'm running a pretty basic setup with the only mods to newnews.pl being the database server information setup.

[fastforward]

Originally posted by Shaman
I'm having the problem that posts made from my board often have no text in the message when viewed on the newsgroup server - except the usenet signature and the tagline from the software.

However, if you type in quite a bit of data (say, 400 or more characters), it will post the message - in my tests, at least.

I'm running a pretty basic setup with the only mods to newnews.pl being the database server information setup.
Shaman, can you check your newnews.pl against mine in the post above.

With so many people reporting the error; there obviously seems to be a problem, but I haven't identified where or why.

[YourHostSucks]

Ok it sems to not feel like working...

I get this

quote:
--------------------------------------------------------------------------------

Connecting to news.*****.***... Connected

Pulling alt.php... No new messages.

--------------------------------------------------------------------------------

It is not getting anything...... ;(

Any Ideas

[YourHostSucks]

*bump* Thanks.... :)

[hacker]

Originally posted by YourHostSucks
*bump* Thanks.... :)

go to www.dyne.com/forums and look for kilobits. He helped me set it up and is very helpful. *nix and Cisco Guru. Say Hacker recommended him!

[Shaman]

telamon# diff /opt/apache/bin/newnews.pl newnews.pl
1c1
< #!/usr/local/bin/perl
---
> #!/usr/bin/perl
5c5
< # $Id: newnews.pl,v 20010820 2001-08-20 23:56:21-05 paul Exp $
---
> # $Id: newnews.pl,v 20010815 2001-08-15 19:15:23-05 paul Exp $
26,29c26,29
< my $db="netforum"; # mysql database name
< my $db_host="xxx"; # mysql host
< my $db_username="xxx"; # mysql username
< my $db_password="xxx"; # mysql password
---
> my $db="database_name"; # mysql database name
> my $db_host="localhost"; # mysql host
> my $db_username="username"; # mysql username
> my $db_password="pasword"; # mysql password
95c95
< # loop through newsgroup and do the business
---
> # loop through newsgroups and do the business
160d159
< db_execute("TRUNCATE TABLE usenet_article_loader");
162a162
> db_execute("TRUNCATE TABLE usenet_article_loader");
168a169,171
>
>
>
1029c1032
< push(@article," > $q\n\n");
---
> push(@article," > $q\n");
1033c1036
< push(@article,"$fbody\n");
---
> push(@article,"\n$fbody\n");
1107c1110
< my $threads = db_fetch("SELECT threadid FROM thread WHERE forumid=$forum
id AND dateline < $purge".(!$config{auto_purge_reg}?" AND regpost = 0":""));
---
> my $threads = db_fetch("SELECT threadid FROM thread WHERE forumid=$forum
id AND isusenetpost <> 0 AND dateline < $purge".(!$config{auto_purge_reg}?" AND
regpost = 0":""));
1114c1117
< if($#post_batch > 500) {
---
> if($#post_batch > 1000) {

[DarkReaper]

I just noticed I have a lot of orphan replies...how does a reply get "lost" and how can I prevent this from happening?

[fastforward]

Originally posted by DarkReaper
I just noticed I have a lot of orphan replies...how does a reply get "lost" and how can I prevent this from happening?
This happens when a post is pulled from the news server that is in reply to a post that either:

a) no longer exists on the news server
b) was rejected by the news server for spam or something
c) was rejected by your spam settings
d) was rejected by your message length settings
e) contained no body
f) some other unexplained reason :)

[DarkReaper]

There's one other problem: It doesn't work with the "who's posted" feature. When I click on the reply count for a thread it says the thread starter has all the posts!

[fastforward]

Originally posted by DarkReaper
There's one other problem: It doesn't work with the "who's posted" feature. When I click on the reply count for a thread it says the thread starter has all the posts!
Does that feature work with any posts made by guests? The userid of all usenet posts are set to 0, ie. Guest.

[DarkReaper]

I guess that would make sense. I don't allow guest posting so I didn't know :)

[DarkReaper]

I remember something about allowing users to have a different Usenet signature, but can't find this in the install files...

[YourHostSucks]

Ok it sems to not feel like working...

I get this

quote:
--------------------------------------------------------------------------------

Connecting to news.*****.***... Connected

Pulling alt.php... No new messages.

--------------------------------------------------------------------------------

It is not getting anything...... ;(

Any Ideas
-------------------------------------------------------

*yawn*

[fastforward]

Originally posted by DarkReaper
I remember something about allowing users to have a different Usenet signature, but can't find this in the install files...
It's in the control panel. You just create a custom profile field for the sig using the normal vB controls, then enter the name of that field in the usenet control panel. Same method applies to the custom email.

[fastforward]

Originally posted by YourHostSucks
Ok it sems to not feel like working...

I get this



quote:
--------------------------------------------------------------------------------

Connecting to news.*****.***... Connected

Pulling alt.php... No new messages.

--------------------------------------------------------------------------------



It is not getting anything...... ;(

Any Ideas
-------------------------------------------------------

*yawn*
If there are no new messages, how can it get anything?

[DarkReaper]

Originally posted by fastforward

It's in the control panel. You just create a custom profile field for the sig using the normal vB controls, then enter the name of that field in the usenet control panel. Same method applies to the custom email.

I did this and it used the default sig instead of my usenet sig.

[YourHostSucks]

O come on, lets not be rude... sheesh, you know there are messages... If I use my news reader I get alot.. *sigh*
But from what you say no news groups ever have messages? (lol)

Ok then.. Nevermind......

[fastforward]

Originally posted by DarkReaper


I did this and it used the default sig instead of my usenet sig.
Make sure the Include user signature on outgoing posts?' is set to YES.
As long as that is set and the 'Custom profile field title for usenet email address' is correct, it should work. It works fine on mine. There are no code edits required to get this to work.

[fastforward]

Originally posted by YourHostSucks
O come on, lets not be rude... sheesh, you know there are messages... If I use my news reader I get alot.. *sigh*
But from what you say no news groups ever have messages? (lol)

Ok then.. Nevermind......
I'm not being rude. I'm simply responding in an appropriate way to your *yawns* and *sighs*.

How do I know there are messages? you simply posted output that indicated there were no available messages. I need more information to be able to help you. Is this the only group you are pulling? Do other groups work? How long does it try to download before telling you there are no messages? Which news server is it? Does the message count increase? Have you tried resetting the message count? Are you using the latest newnew.pl? Are you looking at the same server when using your newsreader?

[DarkReaper]

I have them both as they should be...its not working.

[fastforward]

Originally posted by DarkReaper
I have them both as they should be...its not working.
I forgot to mention... the signature on the forum will always show the usual one. It's only the post that gets sent to usenet that has the custom sig on it. It's donle like this for simplicity and to prevent unnecessary vB code changes.

[DarkReaper]

Its showing my forum default sig on the usenet itself through a newsreader.

My ICQ is 34206559...it'll help make things easier :)

[fastforward]

Originally posted by DarkReaper
Its showing my forum default sig on the usenet itself through a newsreader.

My ICQ is 34206559...it'll help make things easier :)
I can't seem to send a file via ICQ. Here's the newnews.pl I'm using. It works for me. Try it and see what happens. If it doesn't work, the problem must lie in the way the custom profile is configured.

(the file has been renamed as .txt)

[DarkReaper]

# $Id: newnews.pl,v 20010729 2001-07-29 00:10:31-05 paul Exp $

That's from my newnews.pl file...guess I'm a bit outdated :)

What's new/fixed in the new file? I noticed there were some problems when using image and url tags when it was sent to the usenet. Is this fixed?

[fastforward]

Originally posted by DarkReaper
What's new/fixed in the new file? I noticed there were some problems when using image and url tags when it was sent to the usenet. Is this fixed?
Possibly :) This file has been updated as I went along, nothing major, just little tweaks here and there. I haven't really kept track. the latest official release is dated 20010820.

[DarkReaper]

It works now...yay :)

[Mark Hewitt]

Does this function in 2.2.0?

[tamarian]

Originally posted by Mark Hewitt
Does this function in 2.2.0?

Yes. I've had no problems running it on 2.2

[ryankbrooks]

Even if I have the email address option set, newnews.pl posts my default email address instead of the users addr.

Are other folks using this option successfully?

I don't want to use a custom field.

TIA,

R

[H2Hosting.com]

When 2.2.0 release will be available? Thanks

[Shaman]

I took the plunge. To install the usenet.php file in the administration, all you need to do is add this:

Right under "makenavselect("Templates","<hr>");"

// *************************************************
makenavoption("Options","usenet.php?action=options","|");
makenavoption("Groups","usenet.php?action=groups","<br>");
makenavoption("Spam Control","usenet.php?action=spam","<br>");
makenavselect("USENET Control","<hr>");

The rest, you'll have to hack out by hand. It's complex. :)

[yulee]

Hello there,

I read the entire thread, yeah I'm a bit surprised with your hardworking (in special fastforward, sure!) and now I have a little question: 2.2.1 compliant?

I have seen your site fastforward, and I have seen you have some microsoft newsgroups there. I'm trying to install it to run some microsoft newsgroups, so I'm wondering how is the experience with them? Any other experience about MS newsgroups?

Thank you very much for your really amazing hack!!

[spideyrdr]

I'm fairly new to linux, and quite new to vB. Trying to get the hack installed, and I think I'm 99% there. The problem I have is when I run the perl script, if there are new posts on the forum I get the following errors:
NNTPERROR: 500 "POST" not implemented; try "help"
followed by
NNTPERROR: 500 "GROUP" not implemented; try "help".

If there aren't any new posts on the forum, I just get the
NNTPERROR: 500 "GROUP" not implemented; try "help"

With the console output on, everything LOOKS ok, ie. it says it is connecting ok, authenticating, etc. I'm guessing this is something stupid on my part, but a search of this forum as well as the 'net hasn't gotten me anywhere in nearly 3 hours.

Could someone kindly lend some advice? I would greatly appreciate it! I am hoping to have the vBulletin server rolled out by 1/1/02, and this appears to be the last major hurdle before moving it to my staging server. Thanks!

[Scrooge]

I'm having trouble. I am installing the current version of the hack, 20010820 with VB2.2.1. I am up to the point to make changes in Options under my control panel, but when I click options I get the following error:

Warning: Offset 0 is invalid for MySQL result index 6 in /home/reefaqua/public_html/forums/admin/db_mysql.php on line 187

In case it matters, line 187 of my db_mysql.php is:
return mysql_data_seek($this->query_id, $pos);
}

Any ideas where I might have screwed up? Clicking the other links on the cp works, spam control and groups, but options doesn't work. Thanks for any help.

[Scrooge]

I found my own solution. After reading all of this thread I had a bright idea and searched the forums for "warning" and "offset". That's where I found there was a previous thread on an older version of this hack. My question was there and was answered, and the fix helped me. Just in case someone else has this and doesn't think to check the old thread it is here

https://vborg.vbsupport.ru/showthread.php?s=&postid=81936&highlight=offset+warning#post81936

[VirtueTech]

I am new to usenet but I want to give my users this experience.

What news server should I use to subscribe too?

Any help would be great. Right now I am a little lost on this. Its all setup and ready to go...just no news server.

[Scrooge]

I really, really love this hack! I also love the support you guys have already offered. This was a difficult install for me, but I have been able to keep the questions to a minimum because I have found most of them answered already. Thanks :)

Hopefully you haven't become bored with this thread already though because I could use a hand. My problem is that anytime I reply to a thread using "New Reply" it never makes it to usenet. Using the "quote" button does work though. Is there a way to fix this? I would think that I need to edit the newreply template in my new Usenet style, but I have no idea what to do with it.

[VirtueTech]

Originally posted by VirtueTech
I am new to usenet but I want to give my users this experience.

What news server should I use or subscribe too?

Any help would be great. Right now I am a little lost on this. Its all setup and ready to go...just no news server.

bump please

[spideyrdr]

I am running a Mandrake 8.1 box that houses my Apache, php, mySQL, and inn news server. The idea was the news server would simply mirror the forum rather than connect to an external news host. I ran into a problem where I would get the following errors running the script: NNTPERROR: 500 "POST" not implemented; try "help" followed by NNTPERROR: 500 "GROUP" not implemented; try "help". It would seem the resolution is to alter the script slightly so that it would tell the inn daemon that it wanted to connect in a reader mode rather than just a dumb connection. To correct this, I added a little code in two places on version 20010820 of the hack:
$c->mode_reader;
I put it just before line 103 and
$con->mode_reader;
I put that just before line 338.
Things are working much better now, although I think I still have a few bugs to iron out between my many servers... what a headache! I don't know if anyone else is running a setup like this, but I figured I would post it here to save someone some time. Thanks for the hack, fastforward...

[exodus]

This hack is super-sweet!

I think I will be installing it in the near future!

Thanks for your hard work in making this hack!

woohoo.

[Scrooge]

My problem is that anytime I reply to a thread using "New Reply" it never makes it to usenet. Using the "quote" button does work though. Is there a way to fix this?

Shameless bump and begging for help. I have notice that when I use "Post Reply" and then run the script, it says that the post was sent to Usenet, but it never appears. Is there a reason Usenet is not accepting it? I have thought of two possible reasons. One, I'm not sure I entered Useragent correctly, the name I entered was forums@reefaquariumguide.com. Two, when I hover over "Post Reply" I notice that it is posting to a thread id that my site uses, not the actual message number that giganews is using. Is one of these the cause?

Virtuetech: You can do a search on google to find a free newsgroup. I use giganews which is a great server, but it is a pay server.

[VirtueTech]

Originally posted by Scrooge
Shameless bump and begging for help. I have notice that when I use "Post Reply" and then run the script, it says that the post was sent to Usenet, but it never appears. Is there a reason Usenet is not accepting it? I have thought of two possible reasons. One, I'm not sure I entered Useragent correctly, the name I entered was forums@reefaquariumguide.com. Two, when I hover over "Post Reply" I notice that it is posting to a thread id that my site uses, not the actual message number that giganews is using. Is one of these the cause?

Virtuetech: You can do a search on google to find a free newsgroup. I use giganews which is a great server, but it is a pay server.

I ended up using usenetserver.com suggested by webhost. Thanks though. :)

Anyways I'm having the same problems writing to usenet as you are. It just doesn't show up at all on usenet. I think its my SMTP server on my server.

Is there any way to use sendmail? :confused:

[fastforward]

The SMTP server won't affect things being sent to usenet. There are several things that may prevent it showing up on usenet:

For anyone having problems with newnews.pl, I've attached the one I am currently using at my site. It's been running against a dnews server without problems for several months.


An invalid email address. As far as I know, the NNTP server just requires an email that 'looks' like a valid email. However, there may be servers that require a working address.
You don't have posting rights on the server
The group doesn't exist on the server you are posting to.

There are other reasons why your post may not appear in the correct thread on usenet, but for it not to show up at all, it's usually one of the above reasons.

I'm currently re-writing the entire hack in php. There are major changes to the way it works:

It will be specifically for vB 2.2.1 and php version 4.06 and above.
I have written my own nntp class that can be used in other scripts. It's virtually a port of the perl one with a few extras.
The news-pulling script will still be ran using a cron so you will need the php executable on your server. It may be possible to run it via the web, but no guarantees initially.
There will be an option to retrieve only headers. These will be searchable as a normal post. When somebody clicks on the post, the script will download the bodies of all messages in that thread and display it as a normal post and thread. This option is best suited for people running the news server on their own servers where they can control how long a post is kept. This may be added after the initial release depending on time.
The posting to usenet will be handled immediately by the 'newthread.php' and 'newreply.php'. The newnews.php will send any posts that may failed the above step, or were since moderated.
The table structures have changed completely. The external usenet tables will be gone. Additional fields in the vB tables will handle everything. This reduces the volume of data and minimizes the chance of missing usenet posts.

There will be no upgrade path from previous perl versions.

[Scrooge]

For anyone having problems with newnews.pl, I've attached the one I am currently using at my site. It's been running against a dnews server without problems for several months.

An invalid email address. As far as I know, the NNTP server just requires an email that 'looks' like a valid email. However, there may be servers that require a working address.

You don't have posting rights on the server

The group doesn't exist on the server you are posting to.

I don't think any of those would cover my problem. Remember, my "quote" reply button works fine, just the "post reply" button does not seem to work. If any of the above was the problem wouldn't the "quote" reply fail as well?

[spideyrdr]

I've got a small problem with USENET posts getting stuck in the news server...

I'm working with a standalone inn news server and it looks like the posts COMING from my USENET server are off by exactly 1. If I post a message to the group then run newnews.pl, I get no new messages on the forum. If I post a second message or reply to the USENET post, then I get the original message, but not that new message. I wouldn't worry about it except I don't see a TON of my users going the USENET route, thus a message posted on USENET could get "stuck" for a while. Posts coming out of the forum end up in USENET fine. Any help would be greatly appreciated!

Thanks!

PS. I had to do another small mod to the perl to get messages to post to the inn news server... I had to set line ~ 1023 to "push(@article, "X-References: $threadid|$postid\n\n");" thus I needed an extra return in there. I'm surprised that only inn servers want that, but oh well.

[interrealm]

I'm running Windows2000 with IIS... is it possible to set this up? All references I've seen are for apache only.

[interrealm]

btt

[fastforward]

Originally posted by interrealm
I'm running Windows2000 with IIS... is it possible to set this up? All references I've seen are for apache only.
It can be done. There are posts in this thread detailing how.

You might want to wait a few weeks until I release the new PHP version that will not require the headache of installing any Perl modules.

[interrealm]

Sounds good to me... i have no problem waiting...

I would love to be a tester if you need one. my board currently sends gets around 1,200 to 1,300 post on the average daily.

[Scrooge]

I'm still having the same problem, post reply does not work but quote reply does. I just noticed that posting a new thread does not go to the usenet either. Anybody have a clue? Could FF's answers above be the cause after all? I still don't think so since posting with quote reply works correctly. Any help is GREATLY appreciated, I've been trying to fix this for a couple of weeks and really want to fix it so I can share Usenet with my users.

[fastforward]

Scrooge,

Did you install the latest newnews.pl I attached a few posts up?
Install that and if it still doesn't work, I'll see what I can do to help.

The newthread and new reply doesn't actually rely on templates for posting. It's only the quote that needs template changes... and that works.

The other two are simply extracted from the database when newnews.pl runs. It does this by checking a flag (isusenet) in the post and thread tables. This can be set -1, 0 or 1. 0 means it's a local post and will never be sent. -1 means it's already gone to usenet, and 1 means it's pending. Do a query on the post table and check if any are still set to 1.

The other point to remember is a reply to a post will only go to usenet if the post to which it is replying has a valid usenet message ID. If the parent post originated on the forum, it won't have this until we re-retrieve it from the news server on the next batch run. Replies will be held until that retrieval is made. This is why the useragent must remain static otherwise we can not identify which posts came from the forum.

Obviously, new threads do not suffer from this limitation as they do not have a parent.

Anyway... install the latest newnews.pl and report back :)

[Scrooge]

That did the trick, why didn't you say to do that before!? Oh, you did :)

Thanks a ton. This is the best hack out there IMHO. Now that everything is working I'm off to share the good news with my users.

[VirtueTech]

First of thanks for the new "newnews.pl" script Paul. Works great!

Currently if my users don't fill in my "usenet signature" custom profile field then the system shows their VB signature.

Is this how it is supposed to work?

Shouldn't it only allow the custom profile field "Usenet Signature" so that it doesn't show the messed up VB sig withthe VB code in it?

Please help.

[fastforward]

Unless the user specifies a new signature. The normal one goes out to usenet (as long as you select 'Include user signature on outgoing posts?' in the control panel).

If you also select 'Custom profile field title for usenet signature' in the control panel, then the user will have the opportunity to override their default sig with a new one specific to usenet.

However, in either case, the BBCode should be stripped before being sent to usenet. Is this not the case? If not, I'll look into it.

[VirtueTech]

It might be...however my signature just looks horrible in usenet.

It would be best if the system only allowed the users to send the custom Usenet Signature only. At least for my site it would. :)

[fastforward]

Originally posted by VirtueTech
It would be best if the system only allowed the users to send the custom Usenet Signature only. At least for my site it would. :)
I can do that quite easily. I'll try and find time today for you.

[Dontom]

short question:
is there an option not to archive x-archive=no posts?
Thank you!
Tom

[fastforward]

Originally posted by Dontom
short question:
is there an option not to archive x-archive=no posts?
Thank you!
Tom
Sort of. The script already does the necesary checking for x-noarchive, but you need to enable it by making a small edit to newnews.pl.

On or around line 404 find the line that reads:if (!$spam) {Change that to:unless ($spam || $xa) {
After that, all posts with the X-No-Archive header will be rejected.
Note: The X-No-Archive header is not part of RFC977.

[Dontom]

cool, thank you for this fast answer!!
Tom

[fastforward]

Originally posted by Dontom
cool, thank you for this fast answer!!
Tom
One thing I should have mentioned. The messages on the console will show these X-No-archive posts as being rejected because they were spam even though they aren't.
I'll add a CP option for this in the next version.

[fastforward]

Originally posted by VirtueTech
It would be best if the system only allowed the users to send the custom Usenet Signature only. At least for my site it would. :)
For a quick fix to allow this now, find on or around line 877 that currently reads:

$signature = $thread->{signature};

and change to:

$signature = "";

This will cause a blank signature to be sent to usenet if a custom one is not set.

[VirtueTech]

Hey thanks but line 877 of what script? :)

[fastforward]

Originally posted by VirtueTech
Hey thanks but line 877 of what script? :)
newnews.pl

[VirtueTech]

Cool Thanks.

[fsfl2k]

FastForward .....

I have not tried this hack yet but the demo site looks great. I have 2.2.1 or I would have already downloaded it.

I read that there was a re-write to PHP going on with the hack.
Any closer to an estimate on when the PHP version may be available :confused:

thanks,
Anthony

[fastforward]

Originally posted by fsfl2k
FastForward .....

I have not tried this hack yet but the demo site looks great. I have 2.2.1 or I would have already downloaded it.

I read that there was a re-write to PHP going on with the hack.
Any closer to an estimate on when the PHP version may be available :confused:

Complete:
The collection of the news and loading into vB (including single part attachments)
The required vB code and template changes.

Not Complete:
The news sending.
Some other options such as custom signatures custom email and some maintenance functionality.

I have been really tied up at work over the past week so I haven't touched it for a while. I'll get the sending part done and release it in the beta forum for a while. It wasn't simply a port of the Perl version, it was a total rewrite.

[fsfl2k]

Great!
Thanks for the update.
I look forward to testing it out when available.

Anthony

[WildWayz]

Hi ya

When I run newnews.pl I get this


Connecting to news.clara.net:119... Connected
Sending authentication info... Authenticated and logged in

Pulling uk.media.dvd...
-> fetching articles 371122 to 371622...
NNTPERROR: 430 No Such Article
NNTPERROR: 430 No Such Article
NNTPERROR: 430 No Such Article
<-- SNIP -->
NNTPERROR: 430 No Such Article
NNTPERROR: 430 No Such Article
NNTPERROR: 430 No Such Article
-> inserting new threads from uk.media.dvd
-> inserted 0 threads
-> finding replies...

Clean disconnection from news.clara.net:119

Expiring usenet articles older than 120 days...
-> uk.media.dvd... OK
Expire completed.


It isn't importing in the posts from uk.media.dvd (a very active newsgroup).

Any ideas what is up?

My forums are at http://www.dvd-critic.net/forums/index.php

James

[fastforward]

The problem is exactly what it says. 'No such article'.

Assuming you haven't changed the article numbers, you'll probably find that your news server provider has reindexed their news spool and renumbered all the articles.

[WildWayz]

How do I find out the article number?

I mean, in groups I have it setup like this

Newsgroup : uk.media.dvd
Destination Forum : uk.media.dvd
Footer :
Enabled : YES
Last Message Number : 0
Servername : default
Username : default
Password : default

I then telnet in and run ./newnews.pl and it updates
Last Message Number : 370200
But no posts are imported.

--James

[fastforward]

Originally posted by WildWayz
How do I find out the article number?

I mean, in groups I have it setup like this

Newsgroup : uk.media.dvd
Destination Forum : uk.media.dvd
Footer :
Enabled : YES
Last Message Number : 0
Servername : default
Username : default
Password : default

I then telnet in and run ./newnews.pl and it updates
Last Message Number : 370200
But no posts are imported.

--James
use a normal news client to check the last number of the message of the last message in the forum. Alternatively you could just set them to 0 and wait for it to catch up. Depends on the size of your batch setting and the number of articles on your news server as to which method is best. Article numbering and this unavoidable problem is explained in the hack instructions.

[WildWayz]

Hi ya

Thanks - got it (I think - it is fetching OK).

I done as you said and looked at the last post to the newsgroup and it's ID was

403318

so I knocked it back to 403000 and it is now downloading them :)

--James

[WildWayz]

Hi ya - another question (sorry)

If I reply to a usenet thread via this hack, it posts it, but if I make a new thread and post it, it doesn't.

Is it meant to be like that?

James

[fastforward]

Originally posted by WildWayz
Hi ya - another question (sorry)

If I reply to a usenet thread via this hack, it posts it, but if I make a new thread and post it, it doesn't.

Is it meant to be like that?

James
Somebody else had that problem recently. I think it was solved by installing the latest newnews.pl. This version wasn't in the full package until just a few days ago. Try this one and see if it fixes it.

[WildWayz]

Hi again

Thanks so much... I got that file and uploaded it, now when I run it I get


Global symbol "$from" requires explicit package name at ./newnews.pl line 787.
Execution of ./newnews.pl aborted due to compilation errors.


Line 787 =

my $from = $vbconfig{bbtitle}." Mailer <$from>";


--James

[fastforward]

Originally posted by WildWayz
Hi again

Thanks so much... I got that file and uploaded it, now when I run it I get



Line 787 =

my $from = $vbconfig{bbtitle}." Mailer <$from>";


--James
Hehe.. I just edited that line before I uploaded it. It was hard coded with my email address. I guess I should have tested it. As a quick fix, just change the '$from' to your email address. You need to add a backslash before the @ sign.

So for me, it looks like:

my $from = $vbconfig{bbtitle}." Mailer <admin\@dbforums.com>";

[WildWayz]

Thanks :)

This is a really cool hack!

--James

[WildWayz]

Hmmm

doesn't look like new posts or replies are being sent to the usenet group.


Posting message by James Kapherr to uk.media.dvd... Sent
Clean disconnection from news.clara.net:119

Expiring usenet articles older than 120 days...
-> uk.media.home-cinema... OK
-> uk.media.dvd... OK
Expire completed.

Says it has sent it ok - but it isn't showing.

Any ideas?

BTW - do u have a paypal account - wanna tip you for your help :)

James

[fastforward]

No need to worry about tipping. :)

Are these threads visible? In the post table is a column called 'visible'. newnews.pl will only send posts that have this column set to one. This means unmoderated posts will not be sent. The other criteria is the post must be made in a forum that exists in the usenet_group table and the usenet group must be turned on in the control panel.

The other obvious thing is set the 'Send posts to usenet?' in the control panel to Yes.


Check those things and I'll see if I can figure out what else might cause it.

[WildWayz]

Hi ya

in user_groups, enabled = 1.
in posts, all posts are set to visable = 1.
send posts to usenet = yes.

The reply to usenet post was working until I installed that newnews.pl u gave me above (but it wasn't posting new topics to the newsgroup).

Since updating, I cannot post new topics or reply to others (in terms of posting them to the NG).

James

[fastforward]

Hmm. You'd better revert back to the old one for now. When I get home this evening, I'll see what I can do.

[WildWayz]

Hi ya

to narrow it down, I have found it to be like this (using the newnews.pl from the 1st page in the tar.gz file).

New Threads = doesn't show in newsgroup
Reply to = doesn't show in newsgroup
Quote = DOES show in newsgroup

So in order for it to post to the newsgroup, it seems I have to click on Quote then submit. Then when newnews.pl updates via cron, it posts it.

Otherwise I get it rejected by the server.

--James

[WildWayz]

Also,
when I hit on quote then reply, the newsgroup reads my posts as a single line paragraph ie no line breaks.

Any way we can get linebreaks to work?

James

[fastforward]

Originally posted by WildWayz
Also,
when I hit on quote then reply, the newsgroup reads my posts as a single line paragraph ie no line breaks.

Any way we can get linebreaks to work?


Line breaks work for me. I have also been unable to reproduce your other error. What platform are you on?

To verify that the script is sending things to usenet, run this query immediately after you post a thread. Substitute '$$newsgroup->{forumid}' with the forumid in which the thread was posted.

SELECT e.csig,
b.email,
b.showemail,
b.signature,
a.threadid,
a.postusername,
a.title,
d.newsgroup
FROM thread AS a,
user AS b,
usergroup AS c,
usenet_group AS d,
userfield AS e
WHERE a.visible=1
AND forumid=$$newsgroup->{forumid}
AND a.postuserid=b.userid
AND b.usergroupid=c.usergroupid
AND b.userid=e.userid
AND a.isusenetpost=0
AND a.forumid=d.forum;

[WildWayz]

Originally posted by fastforward

Line breaks work for me. I have also been unable to reproduce your other error. What platform are you on?

To verify that the script is sending things to usenet, run this query immediately after you post a thread. Substitute '$$newsgroup->{forumid}' with the forumid in which the thread was posted.

SELECT e.csig,
b.email,
b.showemail,
b.signature,
a.threadid,
a.postusername,
a.title,
d.newsgroup
FROM thread AS a,
user AS b,
usergroup AS c,
usenet_group AS d,
userfield AS e
WHERE a.visible=1
AND forumid=$$newsgroup->{forumid}
AND a.postuserid=b.userid
AND b.usergroupid=c.usergroupid
AND b.userid=e.userid
AND a.isusenetpost=0
AND a.forumid=d.forum;


Hi ya

I am on RedHat 7.0, PHP4.1.0, MySQL 3.23.45

When I run the query, it doesn't like e.csig so I removed it and it outputted this....

email showemail signature threadid postusername title newsgroup
james@dvd-critic.net 1 <center><table width="95%" height="10" bgcolor="#3... 353 James@DVD-Critic OT: Test post uk.media.dvd


--James

[WildWayz]

Also, i noticed it doesn't pull in every post and often it misses some out for some reason (even if I purge the tables and re-grab them).

James

[VirtueTech]

When can we expect a PHP version of newnews.pl?

Thanks. :)

[Dontom]

my bbcode replacing doesnt work properly:(
example here:
http://www.mtb-news.de/forum/showthread.php?s=&threadid=11541

I use:


[q1] is replaced by
&nbsp;&nbsp;<font size="1" color="blue">{param}</font>

any idea?
tom

[bitbender]

FF,

Still running great on my board! ! :cool:

Developing my 2.2.1 upgrade now, so, I await your new release in awe with the others!! :up:

This is/has been a great addition to my board, and has helped draw new members, as well as providing fresh material daily for the membership to post on! :supwink:

[CJi]

I can't wait for a 2.2.1 release of this!!

[WildWayz]

doesn't it work on 2.2.1 ?
I have it on mine, but it might explain why there are bugs.

James

[Tim Wheatley]

Damn, I stupidly assumed this worked on 2.2.1. :cry: I've installed it all and was getting nothing...

[WildWayz]

ya - I thought it was gonna work on 2.2.1 too - but it doesn't fully work - I CAN post things to the newsgroup - but ONLY if I click on "Quote". Reply/New Thread = no submission to usenet :)

--James

[Tim Wheatley]

How's it working at the moment? You have to post something, quote it in a reply and what you quote in the reply is posted?

[bitbender]

Y'all, I believe that FastForward said earlier (a few posts ago) that a complete rewrite for VB 2.2.1 was in the works, and will be forthcoming. From his comments, I suggest waiting.

It works well on my current board(2.0.3), but I have not tried to apply it to my upgrade to 2.2.1. I'm sorta waiting until it's available.

I have alot of hacks to reapply to my 2.2.1 build, and from the results of the current usenet hack, i'm more than willing to wait, even if it means staying on 2.0.3 for a while longer. FF has done a totally stellar job on this, and I can't imagine how much better the new one's gonna be!!

The way it works on my board is that when one of my member ship post a reply to a usenet thread, it looks just like any other post.

I have the newnews.pl script setup to execute twice daily. At those times, any replies made to usenet posts are posted back to the usenet server, and newsgroup the thread originated from. When it does this, it issues messages to indicate it has posted a reply, and if email notification is enabled, a email is sent to the forum member that the thread reply was posted to the usenet group server.

After the the newnews.pl script has completed execution, you should be able to go to the usenet newsgroup via Outlook epress (or any news group viewer) and see the reply.

This is the procedure I used to certify that all was in place and working as expected.

(Hope I got it right, FF! :wink: )

Tim, I hope this helps, else, holla at me again!

Bit's

I went and posted a few replies, and ran the script manually, this is what I got! It's a PNG image, so it's very small:
http://www.bitbenderforums.com/ftp/usenet2.png

[GameCrash]

Yes, that will be a great for my C# site as MS provides many newsgroups... when do you think you can give us a vB 2.2.1 version?

[Sanjiyan]

I am gettng this error message: (Please see that attached Image1.jpg to display the error message....)

I am running:

Windows 2000 Server
Apache Win32
PHP 4.1.1 (lastest stable from the php site)
MySQL 3.23.47-nt
ActivePerl 5.6

I have printed off and followed the install instructions to the letter and did the PPM install of the relvent packages but I am getting this error message. I would really love to use this usenet hack, and so I am asking everyone here for help.

Please either reply here or email me at sanjiyan@blueyonder.co.uk

I thank you in advance, plus a bog bog bog box of Roses to who ever helps me :D

[Sanjiyan]

I fixed this problem my self, I just deleted Perl and reinstalled it all again.

Now I am getting
NNTPERROR: 430 No Such Article

errors...

looking into that now

[Sanjiyan]

Urm

Quote codes:

6. Usenet quotes can be wrapped in custom bbcodes, according to the level of the quote (up to three levels). You can set the colours and indent levels of the quotes by adding the following bbcodes to your current style set as follows:

[q1] REPLACED WITH &nbsp;&nbsp;<font size="1" color="blue">{param}</font>
[q2] REPLACED WITH &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<font size="1" color="darkred">{param}</font>
[q3] REPLACED WITH &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<font size="1" color="green">{param}</font>

This is mentioned int he install.txt file yet I still dont understand or know where do I put this text so it works?? just I cant find where to dump it, any one got any ideas?? Of course you have, alot of you arent a newbie at this like me and are experienced??

[bitbender]

One of the things one needs to do is to browse the news group, and collect the starting numbers for the articles. Not providing a starting point did not seem to work well for me.

Once I gave an article# starting point for each of the news groups I was extracting from, things began to flow much smoother. The article# can be extracted from the header information of a message. I used Outcrook Express, subscribed to each newsgroup, retrieved new messages for each of the news groups, and then sorted by received date, and then used today-1 message's to extract my article numbers, via the properties button, once you open an individual news item.

As for the Perl Reinstall, I had to do that also. I found an excellent procedure for an integrated Apache/Perl distribution for Win32, and that procedure turned out to be a working combination for my board, and W2K .

Sanjiyan

Add those to your BB codes section under the Amin panel. Then, when the posts are displayed, you can see them. They work great on my board!

[Sanjiyan]

Originally posted by bitbender
Sanjiyan

Add those to your BB codes section under the Amin panel. Then, when the posts are displayed, you can see them. They work great on my board! [/B]

I've just tried this, but it does not seem to work, all it says is:

is replaced with ()
[edit] [remove]

is replaced with ()
[edit] [remove]

is replaced with ()
[edit] [remove]



Am I being thick?

Sanjiyan

[webhost]

Been running this script for months, then this morning I get this error.

Database error in vBulletin: Invalid SQL: SELECT COUNT(*) AS posts FROM post WHERE isusenetpost=0
mysql error:
mysql error number:
Date: Tuesday 22nd of January 2002 07:00:44 AM
Script: /vbulletin/index.php3
Referer:

Any ideas anyone?

[interrealm]

Just checking in on the status of the new version? that won't require all the perl mod's.

[Tim Wheatley]

Still wondering when a nice shiny and new 2.2.1 version may turn up? :D

[Frank]

2.2.2 now ;)

[Tim Wheatley]

Yeah -- what he said. ;)

[Mark Hewitt]

So this definitely does not work on vbulletin 2.2.1/2 then?

I agree a totally PHP version of the script would be great. As the reason I had to remove the newsgroups was that I moved server and couldn't do all the Perl module installation etc.

I had originally planned the newsgroup part of my site to be off for about 2 weeks, now 6 months later my members have forgotten about it completely!

(So just as long as it doesn't require modifying PHP!).

[WildWayz]

Originally posted by Mark Hewitt
So this definitely does not work on vbulletin 2.2.1/2 then?

I agree a totally PHP version of the script would be great. As the reason I had to remove the newsgroups was that I moved server and couldn't do all the Perl module installation etc.

I had originally planned the newsgroup part of my site to be off for about 2 weeks, now 6 months later my members have forgotten about it completely!

(So just as long as it doesn't require modifying PHP!).

its a bit iffy when using 2.2.1 - basically, it can spool in the posts, but it is hit and miss whether replies/new posts to it via the forums go through or not.

Also, it stops picking up posts from usenet if I use the same name to post as my forum name.

--James

[Mark Hewitt]

OK. Can we contact vb.org people to get the tag for this changed to vb2.0.3 then. And then when (if) a 2.2.2 version comes out a new thread can be started.

[Erwin]

A PHP version would be an EXCELLENT idea. I waith with anticipation...

[Dontom]

Hi,
i installed all neccessary modules, but perl wont find them, do i have to reboot before??

this is what i get:
Can't locate News/NNTPClient.pm in @INC (@INC contains: /usr/lib/perl5/5.6.0/i386-linux /usr/lib/perl5/5.6.0 /usr/lib/perl5/site_perl/5.6.0/i386-linux /usr/lib/perl5/site_perl/5.6.0 /usr/lib/perl5/site_perl .) at ./newnews.pl line 42.
BEGIN failed--compilation aborted at ./newnews.pl line 42.

I am running perl 5.6.1 - so where would I change the @INC??
Thank you very much!
Tom

[Tim Wheatley]

That means that the NNTPClient perl module isn't installed in the perl dir (doesn't it?)

[bitbender]

Yep, thas what it means...The module package did not install cleanly. That is the current impasse I am fighting thru.

I had downloaded Perl5.6.1, and was using that on my test server. to date, I have been quite unable to get the modules to install using MCPAN.

AS I tell my team at work, "It's a work in Progress" :o

[FlyBoy73]

Is anyone actively working on the compatibility issue? I am also running 2.2.0 and would really like to have this hack..

David

[Erwin]

I'm running 2.2.2 and would love to install this, but I'm holding back because other members seem to indicate it does not work for 2.2.x

[Gilby]

Originally posted by dFrisk
Is anyone actively working on the compatibility issue? I am also running 2.2.0 and would really like to have this hack..

David

I have written a php version (that works on vB 2.2.2, and possibly earlier versions), a complete rewrite of the hack (tailored to my needs) that I just need to make portable and then package it up. Hopefully I'll get that available tonight for everyone.

[dost]

Originally posted by Gilby
I have written a php version (that works on vB 2.2.2, and possibly earlier versions), a complete rewrite of the hack (tailored to my needs) that I just need to make portable and then package it up. Hopefully I'll get that available tonight for everyone. Wow, that's wonderful! :D Does it mean that it won't require any perl modules? And I hope that it will work on vB 2.0.3, too :cry:

[Sanjiyan]

Originally posted by Gilby


I have written a php version (that works on vB 2.2.2, and possibly earlier versions), a complete rewrite of the hack (tailored to my needs) that I just need to make portable and then package it up. Hopefully I'll get that available tonight for everyone.

if it dont need perl then you are a god send and I shall shower you in boxes of roses as a thank you :)

so when you gonna make this unoffical updated hack available ;p

come on hurry it up, you have us all waiting now you know.

[bitbender]

Doesn't need Perl?, what will I do ?? I've spent days mastering a miniscule knowledge of the subject, and now, it's all for naught??

I can't wait to see one without the Perl !!


Ohh, and we'll send you a truckload of Texas BBq ribs !!

[Sanjiyan]

Originally posted by bitbender
Doesn't need Perl?,

I seriously hope it dont, cos I cant run cron from my host, nor any other without payng mega'quids for, considering I onyl want to pull posts from a moderated group that posts maybe 20 posts a day :P

[Gilby]

No perl is needed. It's all in PHP. Athough it does require PEAR since it uses a (modified) NNTP class from that. PEAR should be available on most PHP installations, but I can probably change the code to not require it, so if that requirement happens to be a problem, then it's not hard to change it.

[Gilby]

The PHP version of the NNTP gateway is now available at https://vborg.vbsupport.ru/showthread.php?threadid=35247

[Shaman]

This is/was a great hack. I hope that FF won't let it go by the wayside or that Jelsoft will take it (or the PHP version) on.

[bitbender]

Well, I got my perl module issues sorted out, and my version of Fast Forwards usenet interface appears to be working as designed on vb 2.2.2!!!!

I have only done medium level testing so far. I have placed a one reply in each usenet group, and the cron (W2k Sched task in my case) newnews.pl sorted it right out, send the replies out, and I verified them via M$oft's OutCrook Express's News Viewer!

I changed nothing, just reworked the mods into my 2.2.2 base. Ohh, I DID get the latest newnews.PL from this thread!

All appears well, I'll keep reporting back my results!!

[Shaman]

This is a better hack than the PHP version still. It's more intelligent, mainly. I have it working with 2.2.2 and am loathe to change. :/

[fastforward]

I made a start with a PHP version of this, but as Gilby has already released a version, there's no point continuing it.

I feel more comfortable developing a Perl version anyway as I know a bit more about it that PHP. :)

So, if there is still interest, I will go back to developing the Perl version.

A few things I intend to do:

Reduce the number of modules required
Support for attachments (probably only incoming ones)
Clean up of multipart posts from Outlook and Netscape readers
More options to handle messages regardless of whether it's parent is present
Better performance with less server load. This is especially a problem with mailing list archives that have to use the title for threading.
Extra options to handle x-noarcharve posts such as deleting after a specified time period.
Modularize the script to ease the transfer of moving over to vb 3 when it arrives. The threaded model of version 3 should make things a lot easier.

A few things I'd like to do if possible:

Clean up the code in the control panel ( volunteers are welcome :) )
Option to recieve posts via a mail gateway such as news2mail.com. This will eliminate the need for a usenet account and will mean no more worrying about message number tracking. The problem will be finding a way to post back to usenet.
Option to retrieve headers only and automagically connect to usenet to retrieve the body when viewed. This will save a boat load of space, but obviously won't allow searching of bodies. This will require hacking the vB code substantially and may not be practical, so don't hold your breath. :)

If people are still interested in a Perl version and anyone has any requests, let me know here.

PS to bitbender: I have been running it on 2.2 on for several weeks with no problems so far.

[Shaman]

I am still interested in it, for sure. I am using it with 2.2.4 now, it was a bit laborious hacking it in by hand, but I did it in under 20 minutes. It all seems to be working perfectly.

I'm not sure about the news2mail gateway but if you can make it work worth our while, that sure makes sense.

Performance isn't bad in my application but I have a local news server and I only track about five groups. I appreciate the work you did!

[bitbender]

Originally posted by fastforward
If people are still interested in a Perl version and anyone has any requests, let me know here.

PS to bitbender: I have been running it on 2.2 on for several weeks with no problems so far.
FF, please count me in.

Yes, I too, have had no issues with it, working just fine, at 2.2.3!

[AntiOnline]

Greetings:

FF, count me in too! This has become a major feature on our site, and you did an incredible job.

We have 75,000 users and growing, 27 hosted newsgroups, and everything's stood up to the test of time!

[jOOPeON]

Hi,

I just started to work again with usenet hack. I used it in
one forum a year ago, but then there was few problems.
Now everything works almost too smoothly :)
One humble question came in my mind...
Is it possible to attach file link to ougoing usenet post, if there is a attachment in post ?

-jOOP

Sample:

Test Test...


File in forum:
http://localhost/forums/attachment.php?postid=2878

--
Signature

[fastforward]

Originally posted by jOOPeON
Is it possible to attach file link to ougoing usenet post, if there is a attachment in post ?
Originally I had no intention of adding support for binary attachments. My reasoning was that you're probably mirroring text groups so have no business posting binary attachments to these groups. ;)

Having said that, in the groups I mirror, people occasioanlly post log files and code snippets which people seem to find acceptable. I do intend to add incoming attachment support, so I guess there's no reason not to add outgoing attachments as well.

[jOOPeON]

Originally posted by fastforward

My reasoning was that you're probably mirroring text groups so have no business posting binary attachments to these groups. ;)



True,

But I like to add just link to file, so the usenet readers could visit the site to see the file.
Actually it could look just as the attachment link in vb posts.

Here in scandinavia, binary groups are going to be removed from the servers, so this trick could be great help especially to buy/sell group users. Now group moderators just tells people to find some free internet space for their file uploads :(

-jOOP

[fastforward]

Ah.. I see. Yeah that could be a nice option, and a lot easier to implement than trying to encode attachments. I'll definately add that.

[jOOPeON]

Originally posted by fastforward
Ah.. I see. Yeah that could be a nice option, and a lot easier to implement than trying to encode attachments. I'll definately add that.

Thank you !

Please let me know when the modification is ready for test.

thanks in advance,

-jOOP

[Gilby]

In the php version of the hack I wrote, I just made a link at the bottom of the post that had the url to the attachment and listed the filename. That method seems to work good and allows my users to post pictures or whatever and it's viewable on the text-only newsgroup as well. Fastforward, you may want to look at how that's done.

[jOOPeON]

Something strange happened today.
One of usenet imports failed to the complete crash of perl.
This happened over and over again, until I noticed that it happens only when importing replys from one of the groups.
When I disabled that group, all another groups imported succesfully. If I enable the "broken" group again, same thing happened.
Newnews just crashes when " loading replies 2" .
I tried to increase the last message number wíthout any help.

What could be wrong?

During translations I'm temporary running localhost, IIS 5, Advanced Server, vB 2.2.3, newest perl and PHP ( going back to linux for sure )

-jOOP

ps. I could include the dump from Windows, but it may not help?

[etom]

Hi.

Just installed this great peace of software :).

Fetching (the first) 500 articles from my newsserver was done in 2 minutes, but inserting threads and loading replies needed 3 hours !!

I want to import 40000 posts from my server to vb, but at this speed ?

What to do or to look for ?

PS: my vb is currently at 700 posts.

Thanks
etom

[fastforward]

Well that obviously doesn't sound right :)

Even with a low spec machine it shouldn't take anywhere near that long. As an example, my single processor PIII750 inserts a batch of 500 into the database in 2 or 3 minutes. This is with just under a million posts.

There are few things you should check.
1) The Perl version and it's modules. The Perl version should be 5.005_03 or higher and make sure you have the latest modules.

2) Make sure all indexes are in place in the database. Critical indexes are on the msgid columns and the refs columns. Also make sure all the original vB indexes are in place.

At what point is the delay occuring? What is displayed on the screen?

[etom]

Shell output:
...
-> fetching 23941... OK
-> fetching 23942... OK
-> fetching 23943... OK
-> processing article batch...
-> requested 500 messages... 0 not available or rejected.
-> inserting new threads from alt.x.y.z
[this needed 2 hours.]
*-> Post 2667 skipped... (too long)
-> inserted 113 threads
-> finding replies...
-> loading 367 replies...
[waiting for 4 hours here (currently at 455 = will finish soon)]

My hardware:
Its a 1000 Mhz AMD and a remote 1400 Mhz AMD Database Server, not much load on both.

No changes to default indexes were made ... do You have a reference structure to compare with ?

PS: Thank You very much for Your fast help :)

etom

[Stephan Whelan]

etom,

I'm almost positive you'll be missing an index somewhere as I had the exact same problem when I first installed the software. I ran with it for around 2 months before figuring out it there was a significant problem!

[fastforward]

Here's what the indexes should be.

POST
Unique
msgid

Non-Unique
isusenetpost


THREAD
Unique
msgid

Non-Unique
isusenetpost

Non-Unique
regpost


USENET_ARTICLE
Primary key
msgid
forum


USENET_ARTICLE_LOADER
Primary Key
msgid

Non-Unique
forum


USENET_REF
Primary Key
msgid
ref


USENET_REPLACE
Primary Key
id

Unique
original


USENET_SETTING
Primary Key
settingid


USENET_SPAM_FILTER
Primary Key
id

Non-Unique
forum
dtm


USENET_GROUP
Primary Key
newsgroup


USENET_OUTGOING
Primary Key
postid

Non-Unique
ref


USENET_SETTINGGROUP
Primary Key
settinggroupid


Unique
expression

[fastforward]

An update on what's happening with the next version:
I've OOPed everything and split everything into modules and classes.

There will be the following modules:
NNTP - A slight extension to the standard Perl module
MySQL - Database functionality
Article - A class for formatting and processing usenet articles
vB224 - An interface to vBulletin, handling inserts, indexing etc
Config - Script configuration class

These modules will all be called from a single script that controls the program execution.

The forum module (vb224.pm) will have a standard interface that will allow different versions of vB (or even other forums) to be used by simply dropping in the correct module. Basicaly, this module will be passed an instance of the Article class.

This new version should still be a drop in replacement to newnews.pl.

I have completed the first cut of all modules except vB224 (although, I'm well on the way to finishing it). I'm away on vacation next week so I won't be working on it until I get back.

[etom]

Ahh. Thats the problem:

> 80ms pings to my (remote) database <

In newnews.pl the function indexpost() executes the following two queries very often (for every word of an article):

db_execute("INSERT IGNORE INTO word (title) VALUES ($word)");
db_execute("INSERT IGNORE INTO searchindex (wordid,postid,intitle) VALUES ($wid,$id,0)");

When I change the code to use extended multi-value INSERT's instead things speed up dramatically :)

@fastforward: Thank You for Your (fast) aid !!

etom

[fastforward]

Actually, that index function used to do multi-inserts, but I changed it when I switched to Gemini tables. Gemini didn't like being sent a big insert list for some reason and with the transaction support, multiple inserts didn't matter that much anyway.

I suppose I should really add an option in the next version to select whether or not you are using transaction safe tables and fully optimize the SQL for both cases.

[etom]

Hi.

Sometimes when fetching a batch, it stops "fetching" and starts printing
...
NNTPERROR: 420 No current article
NNTPERROR: 420 No current article
NNTPERROR: 420 No current article
NNTPERROR: 420 No current article
...
for the rest of the batch.

Inserting (successfully) fetched threads and replies etc. works fine after that, but it seems it's loosing many posts under some circumstances ?!

etom

[fastforward]

They can be safely ignored. The problem is in the accuracy of the first and last post numbers reported by the server. They often lie about it.

The next version will not rely on the reported article numbers so this won't happen.

You can turn off the errors by finding this line in newnews.plmy $con = new News::NNTPClient($server,$port) or die "Unable to connect to $$newsgroup->{server}";and replacing it withmy $con = new News::NNTPClient($server,$port,0) or die "Unable to connect to $$newsgroup->{server}";
This line is in the connect_usenet function on or around line 336

[etom]

Sure ?

This happens mid of a download:

* last post number on server ~46000
* ~15000 articles on newsserver
* batch running for articles 4000 to 4500
* stoped fetching at 4300 with this 420 error
* next batch starts at 4500 !!

There is missing 4300 to 4500 then, isn't it ?!

PS: @fastforward: your fast support is great :)

etom

[fastforward]

Originally posted by etom
Sure ?

This happens mid of a download:

* last post number on server ~46000
* ~15000 articles on newsserver
* batch running for articles 4000 to 4500
* stoped fetching at 4300 with this 420 error
* next batch starts at 4500 !!

There is missing 4300 to 4500 then, isn't it ?!

Have you determined whether the next batch really starts at 4500? The lastmsgid is entered into the database once an article was successfully retrieved. All those 'no such article' messages mean the lastmsg column is not updated. Thus, the next batch will start at 4300 or 4301.

[george_proost]

Paul, I'm with you all the way.

The Perl script is great. If you need any testing done etc .. lemme know. I've been using your USENET hack for quite a while now. No problems across versions etc...

I have however made some changes for my purposes.

Namely...
To allow for multiple concurrent runs. I use up to 6 different news services. I provide a parm to the newnews module(s) hardcoded and have a perl module for each news reader.

I update PID files and kill the newname if already running.
Just before termination I do a wait of 'x' minutes and then it relaunches itself. (the 4 crons per day make sure there is continuity)

---

Another nice feature would be to allow for grouping of newgroups by eg.:subject JOBS, PGMLANG, OS etcc

and 'ageing/ pruning' of threads based on the group above or specified per newsgroup in days.

eg: i would like to prune JOBS at 14 days while keep PGMLANG forever (or until the MySql limit :) )

If I can help ... i'll be pleased to oblige.

thx

[fastforward]

Originally posted by george_proost
To allow for multiple concurrent runs. I use up to 6 different news services. I provide a parm to the newnews module(s) hardcoded and have a perl module for each news reader.

Do all 6 ever run at once? You must have a badass server! If my two sites happen to update their news at the same time, my poor little machine nearly grinds to a halt! :) The next version will pull the articles and insert them straight away, rather than batch them all up. The slight delay should reduce the server load considerably.

I update PID files and kill the newname if already running.
Just before termination I do a wait of 'x' minutes and then it relaunches itself. (the 4 crons per day make sure there is continuity)

I've been trying to think of a way to continuously 'stream' the news in like a 'IHAVE' feed and run it as a daemon, but I'm pretty sure there's no way with a standard 'suck' account.

Another nice feature would be to allow for grouping of newgroups by eg.:subject JOBS, PGMLANG, OS etcc

and 'ageing/ pruning' of threads based on the group above or specified per newsgroup in days.

eg: i would like to prune JOBS at 14 days while keep PGMLANG forever (or until the MySql limit :) )

I hadn't thought about grouping, but I do intend to add the per newsgroup expire option.

I don't actually use the auto expire option as it takes too long to empty the searchindex on an individual post basis. We need some a timestamp in the searchindex table, but that will probably more than double the size of the table. Mine is already over 1GB. (Or at least it was until yesterday... I'm experimenting with full text search at the moment, so I emptied it.

If I can help ... i'll be pleased to oblige.

Thanks

[fastforward]

etom,

If you download the latest version from the first thread, it has the multi-insert statement for the searchindex table.

There may a few other little differences in this file as it's the one I use that may get tweaked as I go along. You just need to replace your newnews.pl. If you've made your own edits and just want the indexing change... Just replace the indexpost() function.

[george_proost]

:)
Do all 6 ever run at once? You must have a badass server! If my two sites happen to update their news at the same time, my poor little machine nearly grinds to a halt! The next version will pull the articles and insert them straight away, rather than batch them all up. The slight delay should reduce the server load considerably.

I've seen 4 run concurrently ...

866 1/2 gig 20gb HD .. linux 7.... and so on dedicated only to the forum.

I'll be doing other things soon .. and expect server load increases. I will then upgrade to a dual 1.x Ghz... 2GB system (estimated nov-dec)

I don't actually use the auto expire option as it takes too long to empty the searchindex on an individual post basis. We need some a timestamp in the searchindex table, but that will probably more than double the size of the table. Mine is already over 1GB. (Or at least it was until yesterday... I'm experimenting with full text search at the moment, so I emptied it.


yea ! :( ... but lets expand on this one !!! :)

In the post table you can switch the displaybit off. Thats all you do. Then we know that the parent forum is a usenet forum (by adding a column if needed.. i have already for another purpose) or from the posts table we know if it is a usenet post.. or a certain group post.

The post will not be displayed. Ok! you then determine the the lowest usage times of the day and schedule a delete posts in forums usenet where do not display post is on.

You can maybe tell it to do 50 at a time or whatever... hmm


Thanks

Always a pleasure

:D

[NgtCrwlr]

Greets Paul,

Well I finally got motivated enough to update vB from 2.0.3 to 2.2.5. I also updated to the latest newnews.pl and everything seems pretty sweet except for a little line feed problem.

On both (NewThreads) and (Reply posts) the messages display fine locally, but the posts to USENET lose the linefeed formatting on any line that ends with a single carriage return.

Example:

line 1
line 2
line 3

line 4 (space between line 3 and 4)
line 5


And this is the end double spaced from line 5.

Will display on USENET as:


Line 1 Line 2 Line 3

Line 4 (space between line 3 and 4) Line 5


This should be double spaced from line 5.

Everything else seems to be working with vB 2.2.5 very nicely. You have my vote for continuing the Perl version of the hack. I hope that things are going well on your end.

Cheers!

[ixian]

Fastfoward, I know this hack has been out for a while and you've probably heard this before, but I just wanted to say THANKS. This hack ROCKS. I am in the process of overhauling my entire site, including forums, and made this the first hack I installed on my new vb 2.2.5 installation after I imported all my users, posts, threads, etc over. It's working like a charm in the new forums now (they don't go live until next week).

I suspect my users are gonna go nuts for this hack, so thanks again! BTW, I had zero problems (other than a misconfigured CPAN config.pl, but I fixed that and anyway that isn't the hacks fault:) ) installing this hack - there's something to be said for a fresh vb install.

P.S. I've done a few test posts and I don't see the linefeed problem mentioned above w/ 2.2.5. Maybe it's something with the upgrade from 2.03? I installed 2.2.5 on a new server from scratch.

[www.tourc.ca]

fastforward: I must say this is by far the best hack on my site, and had very few problems that weren't sorted out right away, and the documentation is pretty straight forward. Thanks very sincerley for doing this hack and supporting it so damn well! I'll be sticking with the perl version myself! so far there isn't a problem I couldn't solve with a little help from google!

I'm a little new to cron, and I've read this whole thread (all 32 pages) and haven't found an answer. I do hope you get your benz... you definatley deserve it in my books! :) I've also read tons of pages on using cron, and now understand it!

I've created a crontab (crontab -e) and a job 10,30,50 * * * * perl /home/admin/newnews.pl and so far it works and e-mails the output to me, but it has problems finding some perl modules.

if I run the command perl newnews.pl from the prompt via ssh the script runs beginning to end without a problem! it's perfect, problem is I want to obviously have this automated, and cron's the tool, but it seems it can't find the perl modules when cron runs it!

the first time it ran, I got the following message:
Can't locate News/NNTPClient.pm in @INC (@INC contains: /usr/lib/perl5/5.6.0/i386-linux /usr/lib/perl5/5.6.0 /usr/lib/perl5/site_perl/5.6.0/i386-linux /usr/lib/perl5/site_perl/5.6.0 /usr/lib/perl5/site_perl .) at /home/admin/newnews.pl line 42. BEGIN failed--compilation aborted at /home/admin/newnews.pl line 42.

somewhere else in this thread (as I assumed from the message) it states that it can't find the path. I thought it was a little strange that it works fine when I run it manually and not through cron, but I uncommented the line in the newnews.pl script that is:

# uncomment the following line if you installed perl modules to a
# non-standard directory
use lib "/usr/local/lib/perl5/site_perl/5.6.1";

note: the NNTPClient.pm was located (locate NNTPClient.pm) in /usr/local/lib/perl5/site_perl/5.6.1/News/NNTPClient.pm

now it can find the NNTPClient.pm file but now I get the following message and the script stops on the line
use MIME::WordDecoder;

Can't locate MIME/Base64.pm in @INC (@INC contains: /usr/local/lib/perl5/siet_perl/5.6.1/MIME/Decoder /usr/local/lib/perl5/site_perl/5.6.1 /usr/lib/perl5/5.6.0/i386-linux /usr/lib/perl5/5.6.0 /usr/lib/perl5/site_perl/5.6.0/i386-linux /usr/lib/perl5/site_perl/5.6.0 /usr/lib/perl5/site_perl .) at /usr/local/lib/perl5/site_perl/5.6.1/MIME/Words.pm line 85. BEGIN failed--compilation aborted at /usr/local/lib/perl5/site_perl/5.6.1/MIME/Words.pm line 85. Compilation failed in require at /usr/local/lib/perl5/site_perl/5.6.1/MIME/WordDecoder.pm line 77. BEGIN failed--compilation aborted at /usr/local/lib/perl5/site_perl/5.6.1/MIME/WordDecoder.pm line 77. Compilation failed in require at /home/admin/newnews.pl line 45. BEGIN failed--compilation aborted at /home/admin/newnews.pl line 45.


i typed 'locate Base64.pm' and it returned

/usr/local/lib/perl5/site_perl/5.6.1/i686-linux/MIME/Base64.pm
/usr/local/lib/perl5/site_perl/5.6.1/MIME/Decoder/Base64.pm

i tried adding another use lib line below the first in the perl script but cannot get it past this error. I used the following format!

use lib "/usr/local/lib/perl5/site_perl/5.6.1";
use lib "/usr/local/lib/perl5/site_perl/5.6.1/i686-linux";

but this isn't working and I continue to get that last message about not being able to find base64.pm

I'm new to most of this and have tried a ton of things and am kind of stuck here!... Please help, and forgive my ignorance!

dave

[fastforward]

I'm back... in case anyone missed me :)

Sorry I've not responded to this thread for a while. I went through one of those phases when you switch focus to other interests. Web stuff seemed to take a back seat. I'm sure we all go through that now and again.

www.tourc.ca, I've responded via PM to your problem. You should be able to add multiple 'use lib' statements to the script.

NgtCrwlr, I seem to remember a similar problem with line feeds way back. It may simply changing all occurances of '\n' with '\r\n' or something similar. I'll look into it.

I still plan on the major script overhaul for vB version 3.

[NgtCrwlr]

Originally posted by fastforward
Sorry I've not responded to this thread for a while. I went through one of those phases when you switch focus to other interests. Web stuff seemed to take a back seat. I'm sure we all go through that now and again.

No problemo. In the last year my interest in the web has certainly taken a much lower priority too (most of the time dead last). So you are not rowing the boat alone. ;)

Cheers!

-Larry
P.S. You have been missed. :cool:

[asarian]

"Upload the newnews.pl script into any directory and chmod 0755 to make it executable."

Any reason why it needs to be chmod 755? It has username + password hardcoded in it. Lets just make it chmod 700, and we will all feel a lot safer. :)

- Mark

System Administrator Asarian-host.org

---
"If you were supposed to understand it,
we wouldn't call it code." - FedEx

[asarian]

If I may be so bold as to make a tiny suggestion, I would like to change the way newnews.pl now brutally kills (kill -9) when a previous instance is running.

For other "new-news" utilities, I use the following code at the beginning of my prog,

open (NEWSLOCK, '>/tmp/new-news.lock') || exit 1;
exit 1 if (not (flock (NEWSLOCK, 6)));

That will try to acquire an exclusive lock, but leave when it cannot immediately do so. That way, when the previous instance has not finished yet, nothing needs to be brutally killed (jn whatever state of update it might be), and instead this ensures no new update begins until the last one ended.

- Mark

System Administrator Asarian-host.org

---
"If you were supposed to understand it,
we wouldn't call it code." - FedEx

[www.tourc.ca]

crontab is working and the hack is running flawlessy. Simply Amazing! I love it.

just a thought on a feature:
having seperate expire days value for each group

this would be great for those groups that get very little posts, but have good information that would be worth saving for later reference, you could have a high expire day value for them, but groups with huge amounts of posts could be set with a lower expire date value!

just a thought!

has anyone run a larger number of groups with this hack and had indexing enabled!

I tested it with indexing enabled and my searchindex table started getting really large, so its disabled for now, but I'm really missing being able to search the newsgroups through my forums.

does anyone have any kind of aprox. statistics of their site and how big their searchindex tables get and how it might effect performance and/or any other issues that might arise!

hmmm.. now the mundane task of sifting through the groups for spam and making spam filters... yeah!... I'm sure I'll be back with some questions regarding this!

great hack fastforward!

if anyone is currently using this hack on their forums, post the link I wouldn't mind taking a look at how you set it up (template wise). I'm running a seperate template as recommended by fastforward, and would like some ideas. (I love the dbforums look).

if you want to see the usenet gateway working on a site other than www.dbforums.com you can see it at www.tourc.com and use the username: testuser with the same password!

cheers!

note: to anyone first installing this hack be warned, when I was installing and searching the newsgroups I came across a group where someone from another forum was trying to install the same hack, he was successfull in doing so, but during his first run of the script not only did he pull all of the news articles, but he posted all of the thousands of threads he had in that forum to the newsgroup. He got flamed pretty bad, and it is affecting the usefullness of his forums on usenet... It's probably a good idea to create a new group for each newsgroup and not use existing forums! just a warning!

[Dmitriy]

I wonder what are the legal issues of copying posts from public newsgroups and posting them on your own website.

Google is doing the exact same thing at http://groups.google.com/ but does it mean that we can do the same thing too or does google have permissions from all the newsgroups that they allow browser-based access to?

[ixian]

Please don't take this as insulting, because it isn't meant to be, but, you need to seriously read up on what USENET is because I can't understand why this would even be a question.

Here's your first hint - who would sue you and why?

[Gilby]

Originally posted by Dmitriy
I wonder what are the legal issues of copying posts from public newsgroups and posting them on your own website.

Google is doing the exact same thing at http://groups.google.com/ but does it mean that we can do the same thing too or does google have permissions from all the newsgroups that they allow browser-based access to?

When someone posts to a usenet newsgroup, they know that it is going all over the internet and is being distributed, archived, and read by others. Making a web based interface to access the content is simply like someone using a client to access the content, and is also like another newsgroup server getting a feed from another usenet server.

When the posts are imported into the forums or are on google groups, the content is referenced as to who the author is of that content, just like you would have citations in a paper or publication that you'd write. Since it is properly cited, it does not violate any copyright laws.

[Gilby]

Originally posted by ixian
Please don't take this as insulting, because it isn't meant to be, but, you need to seriously read up on what USENET is because I can't understand why this would even be a question.

Here's your first hint - who would sue you and why?

As a webmaster, living in a country where people will sue over anything, it's good to look into any legal issues that might cost you a lot if someone were to sue you.

[ixian]

Originally posted by Gilby


As a webmaster, living in a country where people will sue over anything, it's good to look into any legal issues that might cost you a lot if someone were to sue you.

This is true, but this example is taking paranioa a little to far. USENET, as you yourself pointed out, is one of the few remaining free bastions of the net. It's cooperative, distributed nature pretty much rules out this kind of lawsuit.

Now, something I did think of - the way this hack works, you need your own USENET client account. It MAY violate terms of service for you to use your ISP's client account to redistribute posts for others. You can get around this by running your own nntp server, like I do. Helps me cut out the spam too.

[Bro_Joey_Gowdy]

anyone have a .zip version of this ?

and will it work with 2.2.6 ?

[Pilot]

Any chance of a PHP version of this? I have a RAQ4 and I am sure these Perl modules aren't installed and I have no idea how to install them!

[Dioxin]

unfortunately the download file doesn't exit on the server.

can you reupload it?

and what's the difference between the cgi version and the php newsgroup gateway? any recommendations?

[certify]

I am having the same problem with Dioxim.

Anybody have the tar or zip file, please post it here.

Thanks. :)

[NgtCrwlr]

Here ya go... 2.2.4 is what I believe to be the latest version. I'm on my way to catch some zzz's so post here if you have problems downloading.

http://www.makeitsimple.com/download.php

[fastforward]

Sorry I didn't see these posts. I don't seem to get emails from this board anymore. No idea why.

I have been running the new version of this hack on my boards for a couple of months now. It's more robust than the old version and puts less load on the server. It's object oriented as promised and upgrades or addons will be easier.

I still haven't done the indexing routine yet (I don't use it as I use full text indexes). I do intend to do it though. Also the auto expire is not done, but that's easy.

The reason I haven't yet released it yet is that there are huge changes to the database structure and an utomatic upgrade could be be dangerous. The biggest changes are that the message id has been taken out of the post table a new flag colum is added to identify the status . For this reason, I think it would be better to wait until vB version 3 is out as that may require more changes to take advantage of the threading.

However, if anybody wants to empty out their existing usenet posts and start over, I will write the installation instructions this weekend and post it. The current vB module has been tested on versions 2.6, 2.7 and 2.8.

[NgtCrwlr]

Greetings Paul,

Glad to see you're still plugging away at this.

Ironically, last night I decided to check in and see what was up here. After seeing no activity I thought I'd take a shot at fixing the output formatting problems of 2.2.4, and low and behold you show up too.

I'll throw my .02 vote to keep the non-vb3 version around for at least 6 months or more. I have been playing around at the vB3 beta site and from all the discussions I have read and what I've seen so far, it sounds and looks like they have a lot of work to do yet before we will see a usable (public) vB3.

As a usenet article puller your hack works excellent. I seems to me that if the output formatting problems were fixed the 2.2.4 version may be good enough to use as a posting hack too. Last year I pretty much dropped the idea of using this hack for posting to usenet after you told me you really never intend for it to be a posting gateway. So I'm more than thrilled to see you back and I really hope that you can lend a hand at tracking down the last few bugaboo's in 2.2.4.

If you need a working test platform for 2.2.4 you are welcome to use my site. Either tell me what to do or I'll work out a way to give you access.

[GPSNUT]

Fastforward - First, thanks.

I also agree that a Vbulletin ver. 2.x should be around for a while and ver. 3 I don't think will be out too soon. Who knows how the hacks will work with it. ?

I'm still trying to get your hack installed and have yet to get any Usenet newsgroups installed.

Any additional update and install instructions will probably help my "slowness".

Thanks again.

[jOOPeON]

Hi guys,

I have a small problem with my search word table. Seems like
newnews.pl is stripping away all international characters when indexing?
Table is full of words without common scandinavian letters like ? , ? and ?.
Thats why my search doesn't work like it should.
Should i rebuild index once in a while or is it something that I can fix? Anyway all letters are correctly in posts.

-jOOP

Ps. How long it takes to reindex 300.000 posts :)

[GPSNUT]

When threads get inserted into the forum and I go to access them I get

"This person is on your Ignore List. To view this post click [here] "

I don't have a Ignore List setup. I check in UserCP .

Any ideas?

[GPSNUT]

Another strange one.

Database error in vBulletin Control Panel 2.2.5:
Invalid SQL: SELECT msgid,subject FROM usenet_article WHERE ORDER BY dtm ASC LIMIT 1
mysql error: You have an error in your SQL syntax near 'ORDER BY dtm ASC LIMIT 1' at line 1
mysql error number: 1064

[GPSNUT]

Why only NO THREADS being posted to forum?

-> fetching 46322... OK
-> fetching 46323... OK
-> fetching 46324... OK
-> fetching 46325... OK
-> fetching 46326... OK
-> fetching 46327... OK
-> fetching 46328... OK
-> fetching 46329... OK
-> processing article batch...
-> requested 29 messages... 0 not available or rejected.
-> inserting new threads from comp.sys.mac.graphics
-> inserted 0 threads
-> finding replies...
-> loading 1 replies...

[GPSNUT]

Why is this happening?

[fastforward]

I've never seen that problem before.

Does that happen for every post or just usenet posts? What version of vB are you using?

[GPSNUT]

It only happens to Usenet posts in the forums I have created. Nothing special.

Version 2.2.5

I stuck. I look at SQL tables, etc.

I'm thinking I should replace all PHP files in the forums directory.

I'm so thankful you responded. I really want this to work.

[dpajur]

The download link is broken, can someone please have a look at it?

Just thought I'd let you know :)

[fastforward]

Originally posted by GPSNUT
It only happens to Usenet posts in the forums I have created. Nothing special.

Version 2.2.5

I stuck. I look at SQL tables, etc.

I'm thinking I should replace all PHP files in the forums directory.

I'm so thankful you responded. I really want this to work.
It seems this is a known bug with vB 2.2.5 and php 4.2. It has to do with posts by unregistered users. Usenet posts are effectively exactly that. Read this thread for a fix:
http://www.vbulletin.com/forum/showthread.php?threadid=44621&perpage=15&pagenumber=2

[fastforward]

Originally posted by jOOPeON
Hi guys,

I have a small problem with my search word table. Seems like
newnews.pl is stripping away all international characters when indexing?
Table is full of words without common scandinavian letters like ? , ? and ?.
Thats why my search doesn't work like it should.
Should i rebuild index once in a while or is it something that I can fix? Anyway all letters are correctly in posts.

-jOOP

Ps. How long it takes to reindex 300.000 posts :)
This is something I did not take into account. The index routine does indeed strip more than the default vB routine. This was done purposely to eliminate some of the extraneous crap that vB indexes. Obviously that wasn't such a smart move where extended character sets are required. I'll make sure the next version takes it into account.

If you know a bit about regular expressions, it shouldn't be too hard to fix the code.

[GPSNUT]

Fast Forward ... You rule !

Now, I close but I can't seem to understand the following:

sh-2.05$ cd www/forums
sh-2.05$ perl newnews.pl

Connecting to news-remote.speakeasy.net... Connected
Sending authentication info... Authenticated and logged in

Pulling comp.sys.mac.graphics... No new messages.

Pulling rec.bicycles.tech...
-> fetching articles 338237 to 338437...
-> fetching 338238... OK
-> fetching 338239... OK
-> fetching 338240... OK
-> fetching 338241... OK
-> fetching 338242... OK
-> fetching 338433... OK
-> fetching 338434... OK
-> fetching 338435... OK
-> fetching 338436... OK
-> fetching 338437... OK
-> processing article batch...
-> requested 200 messages... 0 not available or rejected.
-> inserting new threads from rec.bicycles.tech
-> inserted 0 threads
-> finding replies...

Pulling rec.bicyceles.racing... No new messages.

Clean disconnection from news-remote.speakeasy.net

Why doesn't it insert the messages. The strange thing is that sometimes it'll insert just ONE thread out of say 200.

[jOOPeON]

Originally posted by fastforward

This is something I did not take into account. The index routine does indeed strip more than the default vB routine. This was done purposely to eliminate some of the extraneous crap that vB indexes. Obviously that wasn't such a smart move where extended character sets are required. I'll make sure the next version takes it into account.

If you know a bit about regular expressions, it shouldn't be too hard to fix the code.

Sorry, my coding sucks without instructions :)
I guess it have something to do with this subroutine?
(or sub wordsonly/sub remove_bb_code)

# index post body
$pagetext =~ s/^\[q[1-9]\]>+.*$//go; # remove all quoted stuff
if (length($pagetext) < 10000) {
my $text = remove_bb_code("$pagetext");
$text = wordsonly("$text");
my @words = split(/\s+/,$text);
my $words_sel="";
foreach my $word (@words) {
if ($word && ((length($word) >= $vbconfig{minsearchlength})) && ((length($word) <= $vbconfig{maxsearchlength}))) {
$word = $dbh->quote($word);
$words_sel .= "$word,";
db_execute("INSERT IGNORE INTO word (title) VALUES ($word)");
}
}
chop $words_sel;
if ($words_sel) {
my $wordids = db_fetch("SELECT wordid FROM word WHERE title in ($words_sel)");
while (my $wid = $wordids->fetchrow_array) {
db_execute("INSERT IGNORE INTO searchindex (wordid,postid,intitle) VALUES ($wid,$id,0)");
}
}
} else {
console(" *-> Post $id skipped... (too long)\n");
}
}

Or this?

sub remove_bb_code {
my $text = $_[0];
my ($bbo,$bbc);
my $bbcodes = db_fetch("SELECT bbcodetag FROM bbcode");
while (my $bbcode = $bbcodes->fetchrow_array) {
$bbo=quotemeta("[".$bbcode."]");
$bbc=quotemeta("[/".$bbcode."]");
$text =~ s/$bbo|$bbc//gi; # easy stuff
}
$text =~ s/&quot;|&lt;|&gt;/ /gsio;
$text =~ s/&amp;|<br>|<(\/)?body>|<p>|<(\/)?html>//gsoi;
$text =~ s/\[size=[0-9]+\]|\[\/size\]//ig; # size
$text =~ s/\[color=(\"\#)?[A-Za-z0-9]+(\")?\]|\[\/color\]//ig; # color
$text =~ s/\[url(=)?(")?//ig;
$text =~ s/(\")?\](.+)\[\/url\]/$2/gi;
$text =~ s/\[email(=)?(\")?//ig;
$text =~ s/(\")?\](.+)\[\/email\]/$2/gi;
$text =~ s/\[font=(\"\#)?[A-Za-z]+(\")?\]|\[\/font\]//ig; # font
$text =~ s/\[list(=)?[1Aa]?\]|\[\/list(=)?[1Aa]?\]//ig; # list
$text =~ s/\[\*\]/ - /ig;
$text =~ s/\[(\/)?code\]//ig;
return $text;
}


Can you briefly tell me what to seek, so I could try to fix it.
Regards,
Joop

[fastforward]

Originally posted by GPSNUT
Why doesn't it insert the messages. The strange thing is that sometimes it'll insert just ONE thread out of say 200.
It will obviously only insert a thread if the message is a thread starter. Out of the 200 messages only a handful will be thread starters. The rest will be replies.

Also, a reply will only be inserted if it's parent exists. Otherwise it's inserted into the usenet_article table until it's parent shows up. When you see 'finding replies...' on the screen, the script is looking in this table for replies to messages that have just arrived.

After you have been ran the script a few times the numbers will look make more sense.

[GPSNUT]

Thank you. Ok. I ran the script and look at this one.

> fetching 279617... OK
-> fetching 279618... OK
-> fetching 279619... OK
-> fetching 279620... OK
-> processing article batch...
-> requested 4999 messages... 0 not available or rejected.
-> inserting new threads from rec.bicycles.racing
-> inserted 1 threads
-> finding replies...

5000 messages and only one thread starter? Takes a little bit of time.

Thanks for your confirmation.

[fastforward]

Originally posted by GPSNUT
Thank you. Ok. I ran the script and look at this one.

> fetching 279617... OK
-> fetching 279618... OK
-> fetching 279619... OK
-> fetching 279620... OK
-> processing article batch...
-> requested 4999 messages... 0 not available or rejected.
-> inserting new threads from rec.bicycles.racing
-> inserted 1 threads
-> finding replies...

5000 messages and only one thread starter? Takes a little bit of time.

Thanks for your confirmation.
Hmmm... it does seem a little strange. How many messages are in your usenet_article table? Everything in that table should have something in the references column. If not, then something is wrong.

[dpajur]

The link supplied for downloading the hack
http://britishexpats.com/download/usenet_gateway.tar.gz
gives a page not found error.

Could someone email it to me at dpajur@pcekspert.com ?

[fastforward]

Originally posted by dpajur
The link supplied for downloading the hack
http://britishexpats.com/download/usenet_gateway.tar.gz
gives a page not found error.

Could someone email it to me at dpajur@pcekspert.com ?

The link was removed as the new version was imminent (as it has been for several months).

However, it seems that it's probably better to leave the old one in place and release the new one with vB3. The new version is a total rewrite and is much more efficient. I'm mirroring over 150 groups with no excessive load on the server. By waiting until vB3, it will ensure the release is as bug free and as optimized as possible.

I'll upload the latest version of the old script this evening when i get home from work.

[JoshFink]

Ok.. Having a bit of trouble with this. Hopefully someone can suggest a solution.

Here is the error I'm getting :

Can't locate MIME/WordDecoder.pm in @INC (@INC contains: /usr/lib/perl5/site_perl/5.6.0 /usr/lib/perl5/5.6.0/i386-linux /usr/lib/perl5/5.6.0 /usr/lib/perl5/site_perl/5.6.0/i386-linux /usr/lib/perl5/site_perl .) at ./newnews.pl line 45.
BEGIN failed--compilation aborted at ./newnews.pl line 45.


I've found WordDecoder.pm but it's in


/usr/local/lib/perl5/site_perl/5.8.0/MIME/WordDecoder.pm
/root/.cpan/build/MIME-tools-5.411/docs/MIME/WordDecoder.pm.html
/root/.cpan/build/MIME-tools-5.411/lib/MIME/WordDecoder.pm
/root/.cpan/build/MIME-tools-5.411/blib/lib/MIME/WordDecoder.pm


Thanks

Josh

[NgtCrwlr]

Josh, try adding the following line to your newnews.pl

use lib "/usr/local/lib/perl5/site_perl/5.8.0";

Add the line right under the first line like this...

#!/usr/bin/perl
use lib "/usr/local/lib/perl5/site_perl/5.8.0";


Not sure it will work, but it's worth a try :)

Cheers!