Usenet gateway

THIS THREAD IS CLOSED!
Posts in here will NOT be answered

This hack has undergone many major changes since this thread was started. Consequently, most of the posts have become dated and of little use. To coincide with the latest major release (20010712), a new one has been started.

See this thread for the latest version and discussion

[fastforward]

You have to create a forum using the vB functions for every usenet group you want. Then you also have to enter the newsgroup and assign it to the vB forum using the usenet control panel page.

If you did that and they're still not showing, make sure the messages you pulled are current and not just hidden from view by the forum settings

Be careful that you assign the correct forum to the newsgroup or you may end up with usenet messages in a real forum!

[Stephan Whelan]

Fastforward,

To confirm my suspicion, try commenting out line 828 of newnews.pl.
It currently reads

find_emails($from, sub { my($e,$orig) = @_; $email = $e->format; } );

Hmm - I commented out that line, there seems to be a slight increase in speed but not really anything significant.

I've been watch what is going on with my server and i'm wondering if it is something to do with the way mysql is setup on my system...as i've said i'm running it on a Cobalt RAQ4i that people in these forums seem to have mixed feelings about...whilst studying what is going on with the mysql processing during the long periods the following processes seem to spend a long time:

INSERT LOW_PRIORITY IGNORE INTO usenet_article_loader SELECT DISTINCT a.newsgroup, a.forum, a.msgid

The state it seems to be in a lot is:

Copying to tmp table

Which leads me to believe that mysql isn't perfoming well. This is strange as I have 512 Mb of RAM with 30 GB HDD - I followed instructions on changing some of the sartup parameters for mysql but nothing really seems to have a great impact.

Any suggestions?

[fastforward]

That query probably will use a fair bit memory as it's joining to the post table. This was added to eliminate the need for a unique constraint on the msgid column in the post table. The unique key had a potential to cause problems with vB upgrades and uninstalling the hack etc.

You can remove the DISTINCT clause from the query. That is causing a sort when it's not really required as we're joing on msgid which should be unique in both tables. See if that stops it creating a temp table.

Also, make sure you have indexes on the msgid column in the post table and the msgid column in the usenet_article table.

The usenet_article msgid should be the PRIMARY_KEY or UNIQUE.

The post msgid should be a non-unique index.

Has the performance always been the same?
If that still doesn't work, you might try removing the LOW_PRIORITY from the statement. It shouldn't make any difference, but you never know.

[Stephan Whelan]

Also, make sure you have indexes on the msgid column in the post table and the msgid column in the usenet_article table.

The usenet_article msgid should be the PRIMARY_KEY or UNIQUE.

The post msgid should be a non-unique index.

OK - I think I hit gold here...I double checked the indexes on those tables and found the post msgid index missing...I added that and performance has increase by about 1000%! Look like that is the major problem - must have missed that during an upgrade.

I didn't do any other changes...i'm also going to reinstate the line you recommended to comment out of newnews.pl.

[fastforward]

I just noticed the Spam Control page in the control panel isn't refreshing properly after the cosmetic changes made.

I've put a revised usenet.php in the package and made the version 2.8.1.

Only the usenet.php changed so you can just download this one file here and overwrite yours.

The code in here is a total mess and really needs sorting out. I still get the 'headers already sent error' ob the spam page, but it works. I'll look at tidying it up later (when I learn some more PHP )

[Mark Hewitt]

I would really like to be able to use this hack on my forums but I'm concerned that it would eat through the bandwidth (data transfer) on my site and cost me a lot of money.

Does anyone who has used it have any figures about how much bandwidth it would use, obviously it's different for different newsgroups but can anyone give me a general idea?

If I could put it on my site and it would only take (say) 1 or 2GB per month then that would be great but if it was to take (say) 20GB that would be impossible for me.

Thanks

[Stephan Whelan]

Mark,

Does anyone who has used it have any figures about how much bandwidth it would use, obviously it's different for different newsgroups but can anyone give me a general idea?

Well, I haven't got any firm figures but here are my general thoughts on this:

- First reaction is not much!
- Biggest drain will be when you do your first batch pull as this is getting a lot of posts for history
- Depends on how many newsgroups you pull...
- ...and how active they are
- Since the perl script ignores Binary Posts (or posts above a certain amount of lines) these are not a drain on your bandwidth

Overall I don't think it should be too bad, but i'm sure others have some sort of analysis on this.

[chrispadfield]

i would have thought hardly anything if you are only pulling a text database. All it you are doing is pulling some newsposts and getting the article index, i would be suprised if you get close to 100MB in a month mirroring 5 or so reasonably active text news groups.

[Mark Hewitt]

Originally posted by chrispadfield
i would be suprised if you get close to 100MB in a month mirroring 5 or so reasonably active text news groups.

Really! I was expecting much more than that!. Outstanding news if it's accurate. And yes I was thinking about doing about 5 text newsgroups.

PS Does this hack work ok on BETA5? (sorry if thats already covered)

[chrispadfield]

it is only an estimation but if you think about it what other data is there going to be? all the data you are pulling is the actual text posts and they go straight into your db. If you were pulling 1GB of data a month the worry would not be the bandwidth problem but the size of your database! I think matt said recently at sitepoint that a 1,300,000 post database was about 1.5GB or something which should give you an indication... no way will you pull anything near like a GB per month i am pretty certain of it.

About 1 more week i recon until i have another go with this, it has changed a lot since i first installed it on v 1.5