THIS THREAD IS CLOSED!
Posts in here will NOT be answered

This hack has undergone many major changes since this thread was started. Consequently, most of the posts have become dated and of little use. To coincide with the latest major release (20010712), a new one has been started.

See this thread for the latest version and discussion

thanks for this, i am so happy now :)

i am going to install and play and see if i can suggest etc things for it but probably wait to open it up to my members until the v2.0 version comes out.

Have to say this is one of the best features out there for vbulletin and bar co-branding and unlimited depth forums beats most of the additions for v2, thanks so much for making it :)

ok problem...

i installed some of the modules using cpan which was ok but one of them some how decided that it wanted to re-install the whole of perl without even asking. This, scared me quite a lot and i quit it which may not have been the best thing to do i don't know.

Anyway, after that little episode i think i did everything but i get this error message:

now is this a problem with Net::NNTP (not being installed?) or the hostname (which is correct, it works in outlook express anyway). Any suggestions?

thanks..

spamkiller.newsfeeds.com is now known as spamkiller.usenet.com. They changed it around a month ago.

The Net::NNTP is part of libnet which may well have been already installed. If you're getting an error saying it can't fnd the host, it probably means the module is OK. It wouldn't have got that far otherwise.

newsfeeds.com appears to have come back, it is working. it was a silly mistake i think the server name went on to to line in the perl script and it didn't like it...

any way, currently going through the downloads so have it working... why did you have to release it now.. it is 4.30a.m here and i can't stop (i am only kidding just so pleased to play with this!)

one question, the "rejected bad header" what is that for? i get quite a few of these, probably 20% of them so far, is it spam or is it something else they are rejected for?

The bad header is usually due to strange character sets or a bad news client being used to post the article. Usually you'll see this in test and binary groups where people may be testing out there own posting programs. 20% seems high. I was getting that sort of number when I was using alt.test. But a real newsgroup only rejects a tiny percentage.

The script actually puts out the rejected header message when it can't find a message id in the header. This is usually due to the reasons above.

ok few things:

1) get these errors once it has finished going through the articles:

2) Posting hangs

It appears the script hangs when trying to send posts back to the newsgroups. tried twice and nothing has happened for over 10 minutes now.

3) Numbers..

is this odd:

Processing group alt.tv.stargate-sg1...2321 messages.

results:

usenet_ref : 6098
usenet_article : 1028
thread : 204
post : 774

seems a bit weird to me

i am going to try a few more newsgroups but any ideas on why posting ain't working?

The uninitialized value messages are because the script is running using STRICT and with the -W switch. It's from all the regular expression testing in the article parsing bit. It basically means its trying to test an empty variable. It can be ignored and if you take of the -w switch it will disappear.

The numbers you show seem ok. The 2321 are the numbers the server reports it has. This is usually slightly high by maybe 25 to 100 messages. The usenet_ref table is the many to many link between articles and replies. Because you are doing the initial history pull you will find a lot of messages left in the usenet_article table because it contains replies to threads that have since expired from the server. Once history is collected you can set the $expire variable to a sensible value and it will gradually reduce as time goes on.

As for the posting problem, it works for me in alt.test. I'll keep looking looking and let you know if I find any problems. What message are you seeing before it hangs? Do you see the "Posting message by $poster to $newsgroup..." message?

The posting problem is an SQL statement problem. I removed some unwanted fields from the usenet_outgoing table but didn't change the statement. Unfortunately I was using select * instead of listing the fields.

In the post_outgoing routine (the last function in the script) change line 281 that reads my '$q2 = db_fetch(...'
to read:
my $q2 = db_fetch("SELECT poster,newsgroup,subject,refs,body FROM usenet_outgoing");
And change the line below it to:
while ( my ($poster,$newsgroup,$subject,$refs,$body) = $q2->fetchrow_array ) {

Sorry about that. The fix is in the package for download.

There's always something I mess up!

ps. There's also another little change you can make to showthread.php that will ensure the posts are ordered correctly everytime. Not essential but I've put it in the package.

[Edited by fastforward on 01-22-2001 at 12:39 AM]

thank you, i noticed when i posted a post to a thread it actually appeared 2nd to last as opposed to last so if it sorts that out, brilliant.

thanks, will apply fixes later today and see how it goes :)

ok couple of things:

I got the new download and got it working. It let me post ok but the post i did sent about 5 messages, some with the headers in which was a bit bizarre. I thought it might be corrpution in the database from old failed attempts to post so started again, and posting worked.

Now:

1) I am getting this a lot

after it finises a run of article download. Any ideas?

2) Worried about numbers of articles/posts coming through.

Can you explain what usnet_ref and usenet_article are for so i can understand. These are my stats

usnet_ref = 16520 entries
usnet_article = 3179 entries

threads = 724
posts = 1265

it just feels that i should have more threads and posts then i do.

Thanks,

I am going to continue playing.

p.s. i still get heavy heavy failed downloads form alt.tv.stargate-sg1 which is strange, it is a prefectly valid newsgroup. alt.tv.farscape however has very very few. I wonder why that is? perhaps if you have a chance you could have a play with that newsgroup.

--

Anyway, it is a pretty amazing hack i really love it just hope these few things can be ironed out :)

cheers

Chris, the numbers you are seeing are fine. The variable messages are just warnings. See my post earlier in this thread about both these issues. If you really want to hide those warnings remove the -w after the perl line at the top of the script.

I'll look into the excessive bad header thing for you.

regards

are thanks, i understand now. I missed that post of yours, only saw your second one and it answered my questions.

Love the hack, thanks for all the work.

sorry to be such a complete pain here.

Ok, i ran the script again (not on cron yet)

and it went through alt.tv.farscape:

but i go to the forum, and there are only two threads now marked new which seem to have anything else in them..

weird i thought. went into myphpadmin type thing and looked at the last records under usenet_article and searched for them (ie bits of their body text) and got no results.

So it does not appear to be a date/time thing (which would explain why there were not coming up the top marked new), it just appears they are not actually getting into the database.

On this point, i understand that usenet_ref is a many to many thing so there will be loads, but what about usenet_article. Should this not be similair to the number of posts? if so, that is my problem as my usenet_article is 3 times bigger than my number of posts.

Sorry about all these questions, bet you regret releasing this now :)

When articles are pulled down from usenet, their order within the thread is calculated and the article is placed in usenet_article. This table is just a temp holding area. At the same time the usenet_ref table is populated with the mesage ids of the articles it refers to.

The next step is to load any messages from the usenet_article table that has no refs (ie. an initial thread starter) into the thread and post tables. Then it deletes the article from the usenet_article and usenet_ref table.

Next it goes through the usenet_article table in conjuntion with the usenet_ref table and loads any messages that refer to an article already in the post table. It will only place it in to the post table if it is in order. For example, if post 1 and 2 are already in the post table and 4,5 and 6 are in the usenet_article table, the articles will NOT get loaded because article 3 is still not available. If you look at the 'ord' column, only posts at the same level or one higher will get loaded. Then it deletes the article from usenet_article and usenet_ref.

As an example for numbers you might expect; I mirror 6 newsgroups and have 2069 in thread, 4463 in post, 1288 messages still in usenet_article and 4463 in usenet_ref. My $expire is set to 14 days. This means any messages left in usenet_article that are over 14 days old will be deleted. Remember, the reason they are still in usenet_article is because the article they refer to is not available on the server or was rejected for spam or something.

The messages not showing up as new, happens due to the nature of usenet and the way messages are propogated. The modification made to showthread.php uses the 'ord' column to sort by first, then the dateline. This means makes the messages appear in the correct order. However, vBulletin uses date to determine whether the message is new. This means not all messages will show up new when they should. You will also run into problems with users not setting their clock correctly. Although the script handles time zones correctly, I often see articles with a future date because the users pc clock is wrong.

The first version I made of this script created a dummy date based on the order of the article. So each time a post was received the time was set to a few minutes after the article it refered to. This avoided the problem but meant you had no way of knowing when the post was really made. Maybe I should go back to this method. What do you think?

Addendum:
Since posting this, I just took another look at showthread.php. I believe it will be quite easy to fix the new post icon issue. I will have to create an additional dummy date field based on the actual date the article is placed into your forums. The real date will remain where it is now. I will modify showthread.php to use the dummy date field when doing anything with usenet forums but use the normal field for other forums. This will also eliminate the need for the original modification in that file. I'll try to make the fix this evening.

regards

[Edited by fastforward on 01-23-2001 at 10:48 AM]

thanks, that really help sort it out in my mind.

Couple of questions on this:

Say you have 1,2,3,4 of a thread which are in the thread and then 5 is rejected for spam, will 6, 7, 8 etc still go into the thread or will they never make it because 5 is missing?

The reason i ask is i know there are some threads that in the newgroup that only have a few posts in them on my forum, and they havn't been updated to it in the forum and wondering if this is perhaps why.



Another problem:

example

http://x67.deja.com/viewthread.xp?AN...&back=clarinet

i hope that url works from deja.
It is the thread stargate DVDs in alt.tv.stargate-sg1
and is a full thread.

Ok, in my forum i have the thread:
http://www.ascifi.net/forums/showthr...p?threadid=656

but only the first two posts. And not really the first two either:

the first post is fine
the second post is only the quote of the first post. All the origina content is missing
None of the other 10 or so posts are there.

now in the newgroup, on the server spamkillers.newsfeeds.com it looks exactly the same as on deja, there definitely, to my mind seems something not working here because:

- it is a new thread, with no posts missing
- the newgroup has all the posts and threads in an identical fassion to deja.com's listing
- the thread is on the forum and one other post but that post appears corrupted.


--

on this point, i realise that the mod is not supported or anything and you have already spent loads of time trying to help me, i can get you access to my sql database for this forum if you want etc but completly understand if you don't have time not a problem....

---

Date sorting.

i think your old method of adding a few mins would be better, in my opinion the ordering of a thread is more important than knowing when the post was posted (especially when it might be wrong anyway). It would perhaps be possible to have another field for "date2" or something and somehow add that in somewhere at the bottom like

"This newsgroup reported this post was posted at blah" not that important really, i think ordering is more important.

and on the whole 1,2 not 3 then not 4 and 5.

perhaps it would be better to have 4 and 5 anyway listed in the thread if it is known 4 and 5 refer to 1 and 2 (i assume so???). then if and when 3 comes along it can be added in to the thread using the date system.

It might be possible to actually create a post saying:

"sorry, this post is unavaliable" in place of post 3. If post 3 then comes along then add it in.

A situation may happen when someone posts post 3 and then deletes it (i think you can do this can't you) or if it gets deleted by someone else for spam or any other reason and it is a shame for one mucked up post to prevent the whole thread being downloaded.

This assumes that post 4 and 5 refer to post 1, 2 and 3 but i think they do as this is the many to many thing you were talking about for usenet_ref i think so it should work.

I am learning this whole usenet system as we go along so sorry if making mistakes.

The only reason a message will not be put into the forum is if the thread starter does not exist.

You are right, there may be occasions when later posts refer to all levels of the conversation. In fact all messages will refer to the thread starter by default. This fact ensures messages are only discarded if the thread starter is not available.

Usenet is threaded. The only way to post to usenet is by responding to a particular article. You should think of a usenet 'thread' as a conversation that consists of many threads. Each article contains the message ids of all other articles within the thread. Each article within the conversation has an 'order' based on the level within the conversation and the order within a particular thread. All messages within any thread will contain the message id of the conversation starter as a minimum.

So a conversation can consist of:
Looking at the above, you will see that messages 1 thru 4 all belong in the same thread. They will each contain the message id of all articles before them. Message 5, however will not contain the message id of 4 as it does not refer to it. Likewise, message 6 will only contain the message id of number 1 in the header, it will know nothing about the other articles in the conversation. But what is important, is the fact that all messages will have at least the message id of the conversation starter in it's header. So, as long as the conversation starter exists, the messages WILL be put into your forum.

but, unfortunatly, they arn't.

have a look at my example from above, i think you may have missed my first post (like i did to you :) ) for an example of where this does not seem to be happening.

Yep I missed your post. I'd swear blind it wasn't there a minute ago! :o

As for the missing body in Erics message; that IS strange. Obviously something is going wrong with one of my parsing statements. Probably a dodgy regular expression. It seems the first line from all posts is being chopped off. Should be an easy fix. I'm sure that doesn't happen on mine though.

Can you confirm that the missing messages are in the usenet_article table? If they are then you can ignore my earlier post as it was obviously bollox! :( If they're not in there then they are being rejected for some other reason.

Either way I'll look into it.

regards

[Edited by fastforward on 01-23-2001 at 12:33 PM]

yup they are in there so... as you say bollox :) lol

at least i know i am not mad now :) i wonder what it is the problem.. strange really.

OK.. if they are there then the fix should be easy. The way it finds which articles to put in the forum at the moment is by using the following query:

SELECT a.forum, a.msgid, a.dtm, a.subject, a.poster, a.body, a.ord, b.threadid, c.ref FROM usenet_article AS a, thread AS b, usenet_ref AS c, post AS d WHERE a.msgid=c.msgid AND c.ref=b.msgid AND b.forumid=$group->{forumid} AND d.msgid=b.msgid AND ((d.ord + 1 = a.ord) OR (d.ord=a.ord))

This must flawed in some way. I was going to do it by looping through the records but I thought I'd be clever by doing it in one go with this query. I'll look into it and find another way of picking the articles. In the meantime, leave the missing articles where they are. When I give you the fix it should just pick them up and move them to the forum.

One last thing, can you look at the 'ord' and 'refs' column in the usenet_article table for the missing posts. Make sure that something is listed in the 'refs' column and let me know what the 'ord' number is. Then do the same for the 2 messages that in the posts table and check the 'ord' column.

ok for message three from

http://x67.deja.com/viewthread.xp?AN...&back=clarinet

now refs are:

so there is something there

and the order is:

2.

this is perhaps weird as article 3 is

Article 1
---> Article 2
--------> Article 3

should it not be order "3" i wonder?


I have sent you acccess to the sql database to webmaster @ yourdomain.com as you turned email of here, i can send it to another email if you would like

2 is correct. The first message is number 0. This indicates the problem definately lies with the query I posted earlier. All the ord column is really is a count of how many references are in the refs column. I'll start fixing it now.

Replace line 162 of newnews.pl that reads:
with this:
I'm still not sure why you are getting the first line chopped off some of messages. I don't seem to have the problem I'll keep looking.

The above fix is now in the package for download.

You'll need to run the newnews.pl half a dozen times or so to catch up and load all the missing posts.

[Edited by fastforward on 01-23-2001 at 03:36 PM]

wehhayyy massive improvement.

old stats:

usnet_article = 3179 entries
posts = 1265

new stats:

usnet_article = 2700 entries
posts = 5948

what is strange is that i have so many more posts but not that many less usenet articles? strange still me thinks, shouldn't it be a 1 for 1 swap?

There are also some still not in there, i will have to explore why and it is perhaps the missing first article thing we talked about.

I will also explore a bit more that missing first line thing and see if it happens again.

Thanks for fixing this, going to get a few of my users test using it now to see what they think.

Hmmm... That does seem very odd. You definately should have a one to one increase/decrease in those tables. Unless of course you just downloaded a whole bunch of other articles that don't have anything to refer to. But that seems unlikely. What is the setting for your $expire variable? Now that you have pulled all the history, you should have it set to 7-14 days. Remember the number needs to be in seconds though.

Do you have to have "allow guest to post" as "on" to be able to use this hack.

ok my new plan is to start again,i will delete posts, threads etc and see how it goes.

dunefreak,

No you don't need 'Guest posting enabled'. The script simply forces the usenet posts into the tables in a way that makes vBulletin think it was a guest that created the post. All normal vBulletin permissions and options work on the usenet forums.

ok slight problem:

Someone posted on the newsgroup, i think 2 messages (maybe 3 as their post count is 3 but only two messages are found when searching by username.. bizarre)

anyway..

in one thread this were the results (3 posted messages)

this is the correct post in the correct thread. this is when we have problems. Next posts:

and next one

i have no idea what is going on here, it appears that messages are being mixed together, and then all posted to the same thread (the thread where the second post was meant to go never went there... strange)...


One final thing this brought up, if someone deletes their own posts it should be deleted from the awaiting to send to usenet table if it has not yet gone.

any ideas what is going on here? It only occurs when more than 1 message goes from the awaiting to send at table at once.

reading through again this is what appears to have happened.

first post --> sent to correct thread correctly.

2nd post combined with 1st post (including headers) --> sent to forum of first thread.

3rd post combined with 1st post and 2nd post (including headers) --> sent to forum of first thread.

not god :)

Ooops! It seems the array that holds the messages for sending is not cleared before it starts reading the next one. I must admit, I never tested it with more than one message in the outgoing table. Here's the fix:

On line 279 of newnews.pl, just after the line that reads
Add the following line:
This is an extra line, NOT a replacement.

Sorry about that.

thanks a lot :)

now assuming the next run works and i don't find any more problems in the actual collection of the posts (let's hope) a few things/improvements for v2.0 (have a rest first after all the help though!)

1. Delete post from table to be sent if user deletes it. This may be because they post in wrong forum etc.

2. Using BPCode i would think to change the colour of quoted stuff
>stuff
>here
>for example
see. Not quite sure how this can be done really. Perhaps adding the bpcode on import would be best???

3. I am going to seperate posts in newsgroups to post on forums in user account, this won't be hard i doubt and will be part of vbulletin as opposed to this hack.

4. using more than one server, i know of some newsgroups i would like to mirror that are only on 1 specific server, would need a choice for these.

5. auto pruning of a guest's user posts and auto deletion of any of these posts upon upload (referenced by a username in a certain newsgroup or by an email address etc.)

6. Hard one this is, email notification of a new newsgroup post to a thread that a member of the forum has posted in.

7. Can variables be passed into the signature i would like a sig like

type thing. Would be quite cool.

anyway that is just a few possibile things i can think of, more i am sure will come :) perhaps add a few of these in (email notification most useful probably) when you do the change for v2.0 (hope that is not too hard, shouldn't be i would not think).

Cheers again for all your help.

chris

What an original and most excellent hack, I look forward to experimenting with it when V2.0 comes out.

Thanks

i now get the following error message referring to the new line

ok to get it to work just comment out the use strict.

It now works but i get this error message:

i doubt this matters though.

Chris,

It looks like you may have forgotten to use the '@' symbol in front 'article' on that new line you added.

Every mention of 'article' in the sub-routine post_outgoing() should be prefixed with @

What exactly do you have on line 296.

The post_outgoing() routine should be this:

even more silly, you made a type and i copied and paste:

oops it should be article :)

i should have noticed that :)

I just received an email from hostro telling me that I was spamming newsgroups. I know this was not from my account. I have accounted for all posts. All I can imagine is somebody downloaded the script and didn't change the $footer variable (which had my site in it). Please... if you have the script and didn't change it, change it NOW.