Usenet gateway

THIS THREAD IS CLOSED!
Posts in here will NOT be answered

This hack has undergone many major changes since this thread was started. Consequently, most of the posts have become dated and of little use. To coincide with the latest major release (20010712), a new one has been started.

See this thread for the latest version and discussion

Yeah newsfeeds.com definately works. I tested it with that.

Well as you can see, the messages are being inserted into the thread table correctly but the threadid is not being returned by the mysql_insertid function. I'm not sure why this is happening. Maybe it's something to do with your mysql setup, I have no idea. But if it happened to you, it will probably happen for others.

I have added another function in there that checks if the threadid is null and if it is, look it up again using the message id. So it will try to use the mysl_insert_id first and if that fails, it will find it itself.

Try that and see what happens. In the meantime I'm going to see what information I can find out about this function at mysql.com.

Hey, it looks like THAT WORKED!!!!!

Can ya believe it?!?!??!

Kind of a weird thing happened on the first import...the forums showed 'xx threads and xx posts', but if you clicked on the forum, the display was empty. Running the script the second time seemed to work!!!!!!!!!!

I'm going to play around a bit more just to make sure, but thank you!!!!! I'll let you know how it goes tomorrow.

Here was another one this morning...I've added a total of 8 newsgroups. The first few are processed fine, but this one keeps bombing out:

Fetching article body 48392... OK
Processing article batch...
Requested 44 messages... 0 not available or rejected.

DBD::mysql::db do failed: You have an error in your SQL syntax near ')' at line
1 at ./newnews.pl line 444.
Query failed (INSERT INTO usenet_ref (msgid,ref,cnt,dtm) VALUES ('<09020102.3649
@hotmail.com>','(none)',0+1,)) at ./newnews.pl line 444.

???? Looks like a new one.

One the plus side, my posts make it to the newsgroups just fine!

You seem to be hitting all those unusual situations that should never happen! This problem is due the xhdr function not being able to determine the date from the message header. I have added a fix for this.

The other thing you uncovered was the '(none)' in the refs column. This means the message had no references and is a thread starter so there is no need to put it in the refs table as it is already in the thread table. It wasn't doing any harm but it was wasting space.

regards
PAJ
--
the latest version of this hack is always in the post of this thread

Just call me troublemaker

I guess it's good and bad. Helps make your hack compatible with more systems, bad in the fact that I've put you through a bunch of hassle over the past couple of days.

I've installed a second test copy of vbulletin that I'm testing this hack with, and have previewed it to the other forum moderators on our site. They are impressed! Once we get the bugs worked out, it'll be great to roll this out to our users (many of which know nothing about newsgroups!)

I'll give the changes a whirl, and let you know how it goes.

Thanks again for your speedy reply!

When I run the script now, it stops after pulling the first group as follows:

Connecting to News.newsfeeds.com... Connected
Sending authentication info... Authenticated and logged in
Getting article batch from alt.test
Fetching headers of articles 849566 to 849665... done

It doesn't try to parse the articles. I don't think it's traffic related, as this happens each time I try it. I have to control-c to abort the script.

Sorry to be a pain...

I've seen this problem occasionally. It has to do with strange characters in the NNTPFrom header field. Once the batch is downloaded part of the processing is to parse this 'from' header to extract the email and real name. If there is something it doesn't like in here it can take forever to process. I've seen it take 10 minutes before. It does come back eventually though. This is very rare but I will try to determine exactly which characters it has trouble with.

Which newsgroup is it stopping on. I'll try the same one for testing. Try just letting it run for a while to see if it frees up.

regards

It's hanging on alt.test I bumped the article number up in the table to bypass the message, and it got much further along (I think it got to the 5th newsgroup), then bombed again:

DBD::mysql::db do failed: You have an error in your SQL syntax near ''wow $10 fo
r each referals and get paid for reading emails make money','skidrow' at line 1
at ./newnews.pl line 448.
Query failed (INSERT IGNORE INTO usenet_article (newsgroup,forum,msgid,dtm,subje
ct,poster,email,refs,body,msgnum,nntpposter,ord,th readid,postid) VALUES ('alt.fi
shing.catfish',8,'<09020102.3650@hotmail.com>',,'w ow $10 for each referals and
get paid for reading emails make money','skidrow1','skidrow1@hotmail.com','(none
)','hay guys check this out this is so cool u can make so much money just by ref
erals and reading emails its owsome its easy money very easy money all u have to
do is take out 5 mins daily and start makeing money \n\n\nhttp://www.inboxcash.
com/$10/referral.asp?id=521016',6731,'skidrow1@hotmail.com ',0+1,0,0)) at ./newn
ews.pl line 448.

Once again, got around this by bumping up the article number in the table. Clearly there is a character problem somewhere that the script doesn't like.

That SQL error is caused by not finding the date in the header again. There are two possible places for the date in the header, one is the 'Date' field and the other is the 'NNTPPostingDate'. Which ones it fills in is dependent on the news server and the posting client.

When I was using newsfeeds.com to test this script I was parsing the header myself to find any one of these. However, I rewrote that part to use the xhdr function as it's more reliable. Since the the rewite I haven't tested it on newsfeeds as I no longer have an account there. So it seems that newsfeeds is not consistent in which header field is used for the date.

I will add more logic to try date first (the client posting date), then if that's not there, use the NNTPpostingDate (the server date).

I'll have the fix done in an hour or so.