The Arcive of Official vBulletin Modifications Site.

About Developer · **Released**: 07-11-2001

Description:

This hack provides a gateway to selected usenet groups. The entire usenet group is mirrored locally. Posts submitted locally to mirrored newsgroups will also be sent out to usenet. A single perl script is executed via cron that collects news and posts any outgoing articles. Incoming messages are parsed against customizable spam filters and threaded correctly before being placed in your forums. By default, no personal information is sent to usenet other than the posters username and whatever you have globally configured as a footer and organization. The option does exist for a user to include a custom email to use for usenet posts. After initial installation (which involves creating some new tables, adding a few columns to existing vB tables, and entering the database connection information to the news script) configuration is completed via the vB control panel.

See it in action at dBforums.com (usenet forums are at the bottom)
Download latest version for vB 2.2.4

Current version is 20020323 for vB2.xx

Current feature list:

correct threading of outgoing posts on usenet
full control panel integration
email notification to usenet replies
emoticon translation into vb icons
vb code removal or conversion prior to posting to usenet
hyperlinked urls in messages
color coded, italicized & indented quotes
vB style quote to usenet style quote conversions for outgoing posts
logging of outgoing posts
support for multiple news servers
support for seperate footers per forum in outgoing posts
multi-language (selectable) handling of quoted MIME printable headers (for all those funny foreign characters)
handling of mailing archive groups
flexible spam control and replacement variable options for incoming and outgoing messages
configurable auto-expire option
option for users to show email address in their outgoing usenet posts (may be different than normal one in profile
option to enable/disable user signatures on outgoing posts.
fully compatible with vB moderation functions

Still to be done:

canceling of messages after they have been sent to usenet via control message in accordance with RFC1036.
binary attachment support
email to PM gateway

Screeshots:
Main Options
More Options
Configuration of Newsgroups
Add New Newsgroups
Spam Control
What an imported usenet article looks like in vB

The latest version of this hack will always be in this first post of the thread.

Before asking questions in this thread, please read all the posts in this thread. If your question goes unanswered, it will probably be because the question has already been answered countless times in this thread.

This hack was created for use on my forums and will only be supported as time permits.

Links to required Perl Modules

etom · #**452** 03-22-2002, 12:06 AM

Ahh. Thats the problem:

> 80ms pings to my (remote) database <

In newnews.pl the function indexpost() executes the following two queries very often (for every word of an article):

db_execute("INSERT IGNORE INTO word (title) VALUES ($word)");
db_execute("INSERT IGNORE INTO searchindex (wordid,postid,intitle) VALUES ($wid,$id,0)");

When I change the code to use extended multi-value INSERT's instead things speed up dramatically

@fastforward: Thank You for Your (fast) aid !!

etom

fastforward · #**453** 03-22-2002, 12:16 AM

Actually, that index function used to do multi-inserts, but I changed it when I switched to Gemini tables. Gemini didn't like being sent a big insert list for some reason and with the transaction support, multiple inserts didn't matter that much anyway.

I suppose I should really add an option in the next version to select whether or not you are using transaction safe tables and fully optimize the SQL for both cases.

etom · #**454** 03-22-2002, 01:34 PM

Hi.

Sometimes when fetching a batch, it stops "fetching" and starts printing
...
NNTPERROR: 420 No current article
NNTPERROR: 420 No current article
NNTPERROR: 420 No current article
NNTPERROR: 420 No current article
...
for the rest of the batch.

Inserting (successfully) fetched threads and replies etc. works fine after that, but it seems it's loosing many posts under some circumstances ?!

etom

fastforward · #**455** 03-22-2002, 01:54 PM

They can be safely ignored. The problem is in the accuracy of the first and last post numbers reported by the server. They often lie about it.

The next version will not rely on the reported article numbers so this won't happen.

You can turn off the errors by finding this line in newnews.pl

Code:

my $con = new News::NNTPClient($server,$port) or die "Unable to connect to $$newsgroup->{server}";

and replacing it with

Code:

my $con = new News::NNTPClient($server,$port,0) or die "Unable to connect to $$newsgroup->{server}";

This line is in the connect_usenet function on or around line 336

etom · #**456** 03-22-2002, 02:12 PM

Sure ?

This happens mid of a download:

* last post number on server ~46000
* ~15000 articles on newsserver
* batch running for articles 4000 to 4500
* stoped fetching at 4300 with this 420 error
* next batch starts at 4500 !!

There is missing 4300 to 4500 then, isn't it ?!

PS: @fastforward: your fast support is great

etom

fastforward · #**457** 03-22-2002, 02:23 PM

Quote:

Originally posted by etom
Sure ?

This happens mid of a download:

* last post number on server ~46000
* ~15000 articles on newsserver
* batch running for articles 4000 to 4500
* stoped fetching at 4300 with this 420 error
* next batch starts at 4500 !!

There is missing 4300 to 4500 then, isn't it ?!

Have you determined whether the next batch really starts at 4500? The lastmsgid is entered into the database once an article was successfully retrieved. All those 'no such article' messages mean the lastmsg column is not updated. Thus, the next batch will start at 4300 or 4301.

george_proost · #**458** 03-23-2002, 11:17 AM

Paul, I'm with you all the way.

The Perl script is great. If you need any testing done etc .. lemme know. I've been using your USENET hack for quite a while now. No problems across versions etc...

I have however made some changes for my purposes.

Namely...
To allow for multiple concurrent runs. I use up to 6 different news services. I provide a parm to the newnews module(s) hardcoded and have a perl module for each news reader.

I update PID files and kill the newname if already running.
Just before termination I do a wait of 'x' minutes and then it relaunches itself. (the 4 crons per day make sure there is continuity)

---

Another nice feature would be to allow for grouping of newgroups by eg.:subject JOBS, PGMLANG, OS etcc

and 'ageing/ pruning' of threads based on the group above or specified per newsgroup in days.

eg: i would like to prune JOBS at 14 days while keep PGMLANG forever (or until the MySql limit

)

If I can help ... i'll be pleased to oblige.

thx

fastforward · #**459** 03-23-2002, 02:59 PM

Quote:

Originally posted by george_proost
To allow for multiple concurrent runs. I use up to 6 different news services. I provide a parm to the newnews module(s) hardcoded and have a perl module for each news reader.

Do all 6 ever run at once? You must have a badass server! If my two sites happen to update their news at the same time, my poor little machine nearly grinds to a halt!

The next version will pull the articles and insert them straight away, rather than batch them all up. The slight delay should reduce the server load considerably.

Quote:

I update PID files and kill the newname if already running.
Just before termination I do a wait of 'x' minutes and then it relaunches itself. (the 4 crons per day make sure there is continuity)

I've been trying to think of a way to continuously 'stream' the news in like a 'IHAVE' feed and run it as a daemon, but I'm pretty sure there's no way with a standard 'suck' account.

Quote:

Another nice feature would be to allow for grouping of newgroups by eg.:subject JOBS, PGMLANG, OS etcc

and 'ageing/ pruning' of threads based on the group above or specified per newsgroup in days.

eg: i would like to prune JOBS at 14 days while keep PGMLANG forever (or until the MySql limit

)

I hadn't thought about grouping, but I do intend to add the per newsgroup expire option.

I don't actually use the auto expire option as it takes too long to empty the searchindex on an individual post basis. We need some a timestamp in the searchindex table, but that will probably more than double the size of the table. Mine is already over 1GB. (Or at least it was until yesterday... I'm experimenting with full text search at the moment, so I emptied it.

Quote:

If I can help ... i'll be pleased to oblige.

Thanks

fastforward · #**460** 03-23-2002, 03:49 PM

etom,

If you download the latest version from the first thread, it has the multi-insert statement for the searchindex table.

There may a few other little differences in this file as it's the one I use that may get tweaked as I go along. You just need to replace your newnews.pl. If you've made your own edits and just want the indexing change... Just replace the indexpost() function.

george_proost · #**461** 03-24-2002, 05:45 PM

Quote:

Do all 6 ever run at once? You must have a badass server! If my two sites happen to update their news at the same time, my poor little machine nearly grinds to a halt! The next version will pull the articles and insert them straight away, rather than batch them all up. The slight delay should reduce the server load considerably.

I've seen 4 run concurrently ...

866 1/2 gig 20gb HD .. linux 7.... and so on dedicated only to the forum.

I'll be doing other things soon .. and expect server load increases. I will then upgrade to a dual 1.x Ghz... 2GB system (estimated nov-dec)

Quote:

I don't actually use the auto expire option as it takes too long to empty the searchindex on an individual post basis. We need some a timestamp in the searchindex table, but that will probably more than double the size of the table. Mine is already over 1GB. (Or at least it was until yesterday... I'm experimenting with full text search at the moment, so I emptied it.

yea !

... but lets expand on this one !!!

In the post table you can switch the displaybit off. Thats all you do. Then we know that the parent forum is a usenet forum (by adding a column if needed.. i have already for another purpose) or from the posts table we know if it is a usenet post.. or a certain group post.

The post will not be displayed. Ok! you then determine the the lowest usage times of the day and schedule a delete posts in forums usenet where do not display post is on.

You can maybe tell it to do 50 at a time or whatever... hmm

Quote:

Thanks

Always a pleasure

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.08126 seconds Memory Usage 2,328KB Queries Executed 25 (?)
More Information
Template Usage: (1)SHOWTHREAD (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (1)ad_showthread_beforeqr (2)bbcode_code (8)bbcode_quote (1)footer (1)forumjump (1)forumrules (1)gobutton (1)header (1)headinclude (1)modsystem_post (1)navbar (6)navbar_link (120)option (1)pagenav (1)pagenav_curpage (4)pagenav_pagelink (1)pagenav_pagelinkrel (11)post_thanks_box (11)post_thanks_button (1)post_thanks_javascript (1)post_thanks_navbar_search (11)post_thanks_postbit_info (10)postbit (11)postbit_onlinestatus (11)postbit_wrapper (1)spacer_close (1)spacer_open (1)tagbit_wrapper Phrase Groups Available: global inlinemod postbit posting reputationlevel showthread	Included Files: ./showthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/functions_bigthree.php ./includes/class_postbit.php ./includes/class_bbcode.php ./includes/functions_reputation.php ./includes/functions_post_thanks.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete showthread_start showthread_getinfo forumjump showthread_post_start showthread_query_postids showthread_query bbcode_fetch_tags bbcode_create showthread_postbit_create postbit_factory postbit_display_start post_thanks_function_post_thanks_off_start post_thanks_function_post_thanks_off_end post_thanks_function_fetch_thanks_start post_thanks_function_fetch_thanks_end post_thanks_function_thanked_already_start post_thanks_function_thanked_already_end fetch_musername postbit_imicons bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete postbit_display_complete post_thanks_function_can_thank_this_post_start pagenav_page pagenav_complete tag_fetchbit_complete forumrules navbits navbits_complete showthread_complete
Messages:

The Arcive of Official vBulletin Modifications Site.

It is not a VB3 engine, just a parsed copy!