Usenet gateway

THIS THREAD IS CLOSED!
Posts in here will NOT be answered

This hack has undergone many major changes since this thread was started. Consequently, most of the posts have become dated and of little use. To coincide with the latest major release (20010712), a new one has been started.

See this thread for the latest version and discussion

ok couple of things:

I got the new download and got it working. It let me post ok but the post i did sent about 5 messages, some with the headers in which was a bit bizarre. I thought it might be corrpution in the database from old failed attempts to post so started again, and posting worked.

Now:

1) I am getting this a lot

Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.
Use of uninitialized value at newnews.pl line 152.

after it finises a run of article download. Any ideas?

2) Worried about numbers of articles/posts coming through.

Can you explain what usnet_ref and usenet_article are for so i can understand. These are my stats

usnet_ref = 16520 entries
usnet_article = 3179 entries

threads = 724
posts = 1265

it just feels that i should have more threads and posts then i do.

Thanks,

I am going to continue playing.

p.s. i still get heavy heavy failed downloads form alt.tv.stargate-sg1 which is strange, it is a prefectly valid newsgroup. alt.tv.farscape however has very very few. I wonder why that is? perhaps if you have a chance you could have a play with that newsgroup.

--

Anyway, it is a pretty amazing hack i really love it just hope these few things can be ironed out

cheers

Chris, the numbers you are seeing are fine. The variable messages are just warnings. See my post earlier in this thread about both these issues. If you really want to hide those warnings remove the -w after the perl line at the top of the script.

I'll look into the excessive bad header thing for you.

regards

are thanks, i understand now. I missed that post of yours, only saw your second one and it answered my questions.

Love the hack, thanks for all the work.

sorry to be such a complete pain here.

Ok, i ran the script again (not on cron yet)

and it went through alt.tv.farscape:

Requesting article 58127 from alt.tv.farscape...Saved
Requesting article 58128 from alt.tv.farscape...Saved
Requesting article 58129 from alt.tv.farscape...Saved
Requesting article 58130 from alt.tv.farscape...Saved
Requesting article 58131 from alt.tv.farscape...Saved
Requesting article 58132 from alt.tv.farscape...Saved
Requesting article 58133 from alt.tv.farscape...Saved
Requesting article 58134 from alt.tv.farscape...Saved
Requesting article 58135 from alt.tv.farscape...Saved
Requesting article 58136 from alt.tv.farscape...Saved
Requesting article 58137 from alt.tv.farscape...Saved
Requesting article 58138 from alt.tv.farscape...Saved
Requesting article 58139 from alt.tv.farscape...Saved
Requesting article 58140 from alt.tv.farscape...Saved
Requesting article 58141 from alt.tv.farscape...Saved
Requesting article 58142 from alt.tv.farscape...Saved
Requesting article 58143 from alt.tv.farscape...Saved
Requesting article 58144 from alt.tv.farscape...Saved

but i go to the forum, and there are only two threads now marked new which seem to have anything else in them..

weird i thought. went into myphpadmin type thing and looked at the last records under usenet_article and searched for them (ie bits of their body text) and got no results.

So it does not appear to be a date/time thing (which would explain why there were not coming up the top marked new), it just appears they are not actually getting into the database.

On this point, i understand that usenet_ref is a many to many thing so there will be loads, but what about usenet_article. Should this not be similair to the number of posts? if so, that is my problem as my usenet_article is 3 times bigger than my number of posts.

Sorry about all these questions, bet you regret releasing this now

When articles are pulled down from usenet, their order within the thread is calculated and the article is placed in usenet_article. This table is just a temp holding area. At the same time the usenet_ref table is populated with the mesage ids of the articles it refers to.

The next step is to load any messages from the usenet_article table that has no refs (ie. an initial thread starter) into the thread and post tables. Then it deletes the article from the usenet_article and usenet_ref table.

Next it goes through the usenet_article table in conjuntion with the usenet_ref table and loads any messages that refer to an article already in the post table. It will only place it in to the post table if it is in order. For example, if post 1 and 2 are already in the post table and 4,5 and 6 are in the usenet_article table, the articles will NOT get loaded because article 3 is still not available. If you look at the 'ord' column, only posts at the same level or one higher will get loaded. Then it deletes the article from usenet_article and usenet_ref.

As an example for numbers you might expect; I mirror 6 newsgroups and have 2069 in thread, 4463 in post, 1288 messages still in usenet_article and 4463 in usenet_ref. My $expire is set to 14 days. This means any messages left in usenet_article that are over 14 days old will be deleted. Remember, the reason they are still in usenet_article is because the article they refer to is not available on the server or was rejected for spam or something.

The messages not showing up as new, happens due to the nature of usenet and the way messages are propogated. The modification made to showthread.php uses the 'ord' column to sort by first, then the dateline. This means makes the messages appear in the correct order. However, vBulletin uses date to determine whether the message is new. This means not all messages will show up new when they should. You will also run into problems with users not setting their clock correctly. Although the script handles time zones correctly, I often see articles with a future date because the users pc clock is wrong.

The first version I made of this script created a dummy date based on the order of the article. So each time a post was received the time was set to a few minutes after the article it refered to. This avoided the problem but meant you had no way of knowing when the post was really made. Maybe I should go back to this method. What do you think?

Addendum:
Since posting this, I just took another look at showthread.php. I believe it will be quite easy to fix the new post icon issue. I will have to create an additional dummy date field based on the actual date the article is placed into your forums. The real date will remain where it is now. I will modify showthread.php to use the dummy date field when doing anything with usenet forums but use the normal field for other forums. This will also eliminate the need for the original modification in that file. I'll try to make the fix this evening.

regards

[Edited by fastforward on 01-23-2001 at 10:48 AM]

thanks, that really help sort it out in my mind.

Couple of questions on this:

Say you have 1,2,3,4 of a thread which are in the thread and then 5 is rejected for spam, will 6, 7, 8 etc still go into the thread or will they never make it because 5 is missing?

The reason i ask is i know there are some threads that in the newgroup that only have a few posts in them on my forum, and they havn't been updated to it in the forum and wondering if this is perhaps why.



Another problem:

example

http://x67.deja.com/viewthread.xp?AN...&back=clarinet

i hope that url works from deja.
It is the thread stargate DVDs in alt.tv.stargate-sg1
and is a full thread.

Ok, in my forum i have the thread:
http://www.ascifi.net/forums/showthr...p?threadid=656

but only the first two posts. And not really the first two either:

the first post is fine
the second post is only the quote of the first post. All the origina content is missing
None of the other 10 or so posts are there.

now in the newgroup, on the server spamkillers.newsfeeds.com it looks exactly the same as on deja, there definitely, to my mind seems something not working here because:

- it is a new thread, with no posts missing
- the newgroup has all the posts and threads in an identical fassion to deja.com's listing
- the thread is on the forum and one other post but that post appears corrupted.


--

on this point, i realise that the mod is not supported or anything and you have already spent loads of time trying to help me, i can get you access to my sql database for this forum if you want etc but completly understand if you don't have time not a problem....

---

Date sorting.

i think your old method of adding a few mins would be better, in my opinion the ordering of a thread is more important than knowing when the post was posted (especially when it might be wrong anyway). It would perhaps be possible to have another field for "date2" or something and somehow add that in somewhere at the bottom like

"This newsgroup reported this post was posted at blah" not that important really, i think ordering is more important.

and on the whole 1,2 not 3 then not 4 and 5.

perhaps it would be better to have 4 and 5 anyway listed in the thread if it is known 4 and 5 refer to 1 and 2 (i assume so???). then if and when 3 comes along it can be added in to the thread using the date system.

It might be possible to actually create a post saying:

"sorry, this post is unavaliable" in place of post 3. If post 3 then comes along then add it in.

A situation may happen when someone posts post 3 and then deletes it (i think you can do this can't you) or if it gets deleted by someone else for spam or any other reason and it is a shame for one mucked up post to prevent the whole thread being downloaded.

This assumes that post 4 and 5 refer to post 1, 2 and 3 but i think they do as this is the many to many thing you were talking about for usenet_ref i think so it should work.

I am learning this whole usenet system as we go along so sorry if making mistakes.

The only reason a message will not be put into the forum is if the thread starter does not exist.

You are right, there may be occasions when later posts refer to all levels of the conversation. In fact all messages will refer to the thread starter by default. This fact ensures messages are only discarded if the thread starter is not available.

Usenet is threaded. The only way to post to usenet is by responding to a particular article. You should think of a usenet 'thread' as a conversation that consists of many threads. Each article contains the message ids of all other articles within the thread. Each article within the conversation has an 'order' based on the level within the conversation and the order within a particular thread. All messages within any thread will contain the message id of the conversation starter as a minimum.

So a conversation can consist of:

1- Conversation starter =  ord 0
   2- Reply to 1 = ord 1
      3- Reply to 2 = ord 2
         4- Reply to 3 = ord 3
         5- Reply to 3 = ord 3
   6- Reply to 1 = ord 1
      7- Reply to 6 = ord 2

Looking at the above, you will see that messages 1 thru 4 all belong in the same thread. They will each contain the message id of all articles before them. Message 5, however will not contain the message id of 4 as it does not refer to it. Likewise, message 6 will only contain the message id of number 1 in the header, it will know nothing about the other articles in the conversation. But what is important, is the fact that all messages will have at least the message id of the conversation starter in it's header. So, as long as the conversation starter exists, the messages WILL be put into your forum.

Originally posted by fastforward
, as long as the conversation starter exists, the messages WILL be put into your forum.

but, unfortunatly, they arn't.

have a look at my example from above, i think you may have missed my first post (like i did to you ) for an example of where this does not seem to be happening.

Yep I missed your post. I'd swear blind it wasn't there a minute ago!

As for the missing body in Erics message; that IS strange. Obviously something is going wrong with one of my parsing statements. Probably a dodgy regular expression. It seems the first line from all posts is being chopped off. Should be an easy fix. I'm sure that doesn't happen on mine though.

Can you confirm that the missing messages are in the usenet_article table? If they are then you can ignore my earlier post as it was obviously bollox! If they're not in there then they are being rejected for some other reason.

Either way I'll look into it.

regards

[Edited by fastforward on 01-23-2001 at 12:33 PM]