Go Back   vb.org Archive > vBulletin Modifications > Archive > vB.org Archives > vBulletin 2.x > vBulletin 2.x Full Releases
FAQ Community Calendar Today's Posts Search

Reply
 
Thread Tools
Details »»

Version: , by fastforward fastforward is offline
Developer Last Online: Nov 2011 Show Printable Version Email this Page

Version: 2.2.x Rating:
Released: 07-11-2001 Last Update: Never Installs: 25
 
No support by the author.

Description:

This hack provides a gateway to selected usenet groups. The entire usenet group is mirrored locally. Posts submitted locally to mirrored newsgroups will also be sent out to usenet. A single perl script is executed via cron that collects news and posts any outgoing articles. Incoming messages are parsed against customizable spam filters and threaded correctly before being placed in your forums. By default, no personal information is sent to usenet other than the posters username and whatever you have globally configured as a footer and organization. The option does exist for a user to include a custom email to use for usenet posts. After initial installation (which involves creating some new tables, adding a few columns to existing vB tables, and entering the database connection information to the news script) configuration is completed via the vB control panel.
  • See it in action at dBforums.com (usenet forums are at the bottom)
  • Download latest version for vB 2.2.4

Current version is 20020323 for vB2.xx

Current feature list:
  • correct threading of outgoing posts on usenet
  • full control panel integration
  • email notification to usenet replies
  • emoticon translation into vb icons
  • vb code removal or conversion prior to posting to usenet
  • hyperlinked urls in messages
  • color coded, italicized & indented quotes
  • vB style quote to usenet style quote conversions for outgoing posts
  • logging of outgoing posts
  • support for multiple news servers
  • support for seperate footers per forum in outgoing posts
  • multi-language (selectable) handling of quoted MIME printable headers (for all those funny foreign characters)
  • handling of mailing archive groups
  • flexible spam control and replacement variable options for incoming and outgoing messages
  • configurable auto-expire option
  • option for users to show email address in their outgoing usenet posts (may be different than normal one in profile
  • option to enable/disable user signatures on outgoing posts.
  • fully compatible with vB moderation functions
Still to be done:
  • canceling of messages after they have been sent to usenet via control message in accordance with RFC1036.
  • binary attachment support
  • email to PM gateway
Screeshots:
Main Options
More Options
Configuration of Newsgroups
Add New Newsgroups
Spam Control
What an imported usenet article looks like in vB

The latest version of this hack will always be in this first post of the thread.

Before asking questions in this thread, please read all the posts in this thread. If your question goes unanswered, it will probably be because the question has already been answered countless times in this thread.

This hack was created for use on my forums and will only be supported as time permits.


Links to required Perl Modules

Show Your Support

  • This modification may not be copied, reproduced or published elsewhere without author's permission.

Comments
  #492  
Old 10-08-2002, 03:51 PM
fastforward fastforward is offline
 
Join Date: Oct 2001
Location: NC, USA
Posts: 399
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally posted by jOOPeON
Hi guys,

I have a small problem with my search word table. Seems like
newnews.pl is stripping away all international characters when indexing?
Table is full of words without common scandinavian letters like ? , ? and ?.
Thats why my search doesn't work like it should.
Should i rebuild index once in a while or is it something that I can fix? Anyway all letters are correctly in posts.

-jOOP

Ps. How long it takes to reindex 300.000 posts
This is something I did not take into account. The index routine does indeed strip more than the default vB routine. This was done purposely to eliminate some of the extraneous crap that vB indexes. Obviously that wasn't such a smart move where extended character sets are required. I'll make sure the next version takes it into account.

If you know a bit about regular expressions, it shouldn't be too hard to fix the code.
Reply With Quote
  #493  
Old 10-08-2002, 06:11 PM
GPSNUT GPSNUT is offline
 
Join Date: Apr 2002
Posts: 10
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Fast Forward ... You rule !

Now, I close but I can't seem to understand the following:

sh-2.05$ cd www/forums
sh-2.05$ perl newnews.pl

Connecting to news-remote.speakeasy.net... Connected
Sending authentication info... Authenticated and logged in

Pulling comp.sys.mac.graphics... No new messages.

Pulling rec.bicycles.tech...
-> fetching articles 338237 to 338437...
-> fetching 338238... OK
-> fetching 338239... OK
-> fetching 338240... OK
-> fetching 338241... OK
-> fetching 338242... OK
-> fetching 338433... OK
-> fetching 338434... OK
-> fetching 338435... OK
-> fetching 338436... OK
-> fetching 338437... OK
-> processing article batch...
-> requested 200 messages... 0 not available or rejected.
-> inserting new threads from rec.bicycles.tech
-> inserted 0 threads
-> finding replies...

Pulling rec.bicyceles.racing... No new messages.

Clean disconnection from news-remote.speakeasy.net

Why doesn't it insert the messages. The strange thing is that sometimes it'll insert just ONE thread out of say 200.
Reply With Quote
  #494  
Old 10-08-2002, 06:27 PM
jOOPeON jOOPeON is offline
 
Join Date: Mar 2002
Posts: 15
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally posted by fastforward

This is something I did not take into account. The index routine does indeed strip more than the default vB routine. This was done purposely to eliminate some of the extraneous crap that vB indexes. Obviously that wasn't such a smart move where extended character sets are required. I'll make sure the next version takes it into account.

If you know a bit about regular expressions, it shouldn't be too hard to fix the code.
Sorry, my coding sucks without instructions
I guess it have something to do with this subroutine?
(or sub wordsonly/sub remove_bb_code)

# index post body
$pagetext =~ s/^\[q[1-9]\]>+.*$//go; # remove all quoted stuff
if (length($pagetext) < 10000) {
my $text = remove_bb_code("$pagetext");
$text = wordsonly("$text");
my @words = split(/\s+/,$text);
my $words_sel="";
foreach my $word (@words) {
if ($word && ((length($word) >= $vbconfig{minsearchlength})) && ((length($word) <= $vbconfig{maxsearchlength}))) {
$word = $dbh->quote($word);
$words_sel .= "$word,";
db_execute("INSERT IGNORE INTO word (title) VALUES ($word)");
}
}
chop $words_sel;
if ($words_sel) {
my $wordids = db_fetch("SELECT wordid FROM word WHERE title in ($words_sel)");
while (my $wid = $wordids->fetchrow_array) {
db_execute("INSERT IGNORE INTO searchindex (wordid,postid,intitle) VALUES ($wid,$id,0)");
}
}
} else {
console(" *-> Post $id skipped... (too long)\n");
}
}

Or this?

sub remove_bb_code {
my $text = $_[0];
my ($bbo,$bbc);
my $bbcodes = db_fetch("SELECT bbcodetag FROM bbcode");
while (my $bbcode = $bbcodes->fetchrow_array) {
$bbo=quotemeta("[".$bbcode."]");
$bbc=quotemeta("[/".$bbcode."]");
$text =~ s/$bbo|$bbc//gi; # easy stuff
}
$text =~ s/&quot;|&lt;|&gt;/ /gsio;
$text =~ s/&amp;|<br>|<(\/)?body>|<p>|<(\/)?html>//gsoi;
$text =~ s/\[size=[0-9]+\]|\[\/size\]//ig; # size
$text =~ s/\[color=(\"\#)?[A-Za-z0-9]+(\")?\]|\[\/color\]//ig; # color
$text =~ s/\[url(=)?(")?//ig;
$text =~ s/(\")?\](.+)\[\/url\]/$2/gi;
$text =~ s/\[email(=)?(\")?//ig;
$text =~ s/(\")?\](.+)\[\/email\]/$2/gi;
$text =~ s/\[font=(\"\#)?[A-Za-z]+(\")?\]|\[\/font\]//ig; # font
$text =~ s/\[list(=)?[1Aa]?\]|\[\/list(=)?[1Aa]?\]//ig; # list
$text =~ s/\[\*\]/ - /ig;
$text =~ s/\[(\/)?code\]//ig;
return $text;
}


Can you briefly tell me what to seek, so I could try to fix it.
Regards,
Joop
Reply With Quote
  #495  
Old 10-08-2002, 06:47 PM
fastforward fastforward is offline
 
Join Date: Oct 2001
Location: NC, USA
Posts: 399
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally posted by GPSNUT
Why doesn't it insert the messages. The strange thing is that sometimes it'll insert just ONE thread out of say 200.
It will obviously only insert a thread if the message is a thread starter. Out of the 200 messages only a handful will be thread starters. The rest will be replies.

Also, a reply will only be inserted if it's parent exists. Otherwise it's inserted into the usenet_article table until it's parent shows up. When you see 'finding replies...' on the screen, the script is looking in this table for replies to messages that have just arrived.

After you have been ran the script a few times the numbers will look make more sense.
Reply With Quote
  #496  
Old 10-08-2002, 11:36 PM
GPSNUT GPSNUT is offline
 
Join Date: Apr 2002
Posts: 10
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Thank you. Ok. I ran the script and look at this one.

> fetching 279617... OK
-> fetching 279618... OK
-> fetching 279619... OK
-> fetching 279620... OK
-> processing article batch...
-> requested 4999 messages... 0 not available or rejected.
-> inserting new threads from rec.bicycles.racing
-> inserted 1 threads
-> finding replies...

5000 messages and only one thread starter? Takes a little bit of time.

Thanks for your confirmation.
Reply With Quote
  #497  
Old 10-08-2002, 11:44 PM
fastforward fastforward is offline
 
Join Date: Oct 2001
Location: NC, USA
Posts: 399
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally posted by GPSNUT
Thank you. Ok. I ran the script and look at this one.

> fetching 279617... OK
-> fetching 279618... OK
-> fetching 279619... OK
-> fetching 279620... OK
-> processing article batch...
-> requested 4999 messages... 0 not available or rejected.
-> inserting new threads from rec.bicycles.racing
-> inserted 1 threads
-> finding replies...

5000 messages and only one thread starter? Takes a little bit of time.

Thanks for your confirmation.
Hmmm... it does seem a little strange. How many messages are in your usenet_article table? Everything in that table should have something in the references column. If not, then something is wrong.
Reply With Quote
  #498  
Old 10-09-2002, 10:12 AM
dpajur dpajur is offline
 
Join Date: Oct 2002
Location: Dublin, Eire
Posts: 2
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

The link supplied for downloading the hack
http://britishexpats.com/download/usenet_gateway.tar.gz
gives a page not found error.

Could someone email it to me at dpajur@pcekspert.com ?
Reply With Quote
  #499  
Old 10-09-2002, 03:04 PM
fastforward fastforward is offline
 
Join Date: Oct 2001
Location: NC, USA
Posts: 399
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally posted by dpajur
The link supplied for downloading the hack
http://britishexpats.com/download/usenet_gateway.tar.gz
gives a page not found error.

Could someone email it to me at dpajur@pcekspert.com ?
The link was removed as the new version was imminent (as it has been for several months).

However, it seems that it's probably better to leave the old one in place and release the new one with vB3. The new version is a total rewrite and is much more efficient. I'm mirroring over 150 groups with no excessive load on the server. By waiting until vB3, it will ensure the release is as bug free and as optimized as possible.

I'll upload the latest version of the old script this evening when i get home from work.
Reply With Quote
  #500  
Old 10-20-2002, 04:56 PM
JoshFink JoshFink is offline
 
Join Date: Nov 2001
Posts: 207
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Ok.. Having a bit of trouble with this. Hopefully someone can suggest a solution.

Here is the error I'm getting :
Quote:
Can't locate MIME/WordDecoder.pm in @INC (@INC contains: /usr/lib/perl5/site_perl/5.6.0 /usr/lib/perl5/5.6.0/i386-linux /usr/lib/perl5/5.6.0 /usr/lib/perl5/site_perl/5.6.0/i386-linux /usr/lib/perl5/site_perl .) at ./newnews.pl line 45.
BEGIN failed--compilation aborted at ./newnews.pl line 45.
I've found WordDecoder.pm but it's in

Quote:
/usr/local/lib/perl5/site_perl/5.8.0/MIME/WordDecoder.pm
/root/.cpan/build/MIME-tools-5.411/docs/MIME/WordDecoder.pm.html
/root/.cpan/build/MIME-tools-5.411/lib/MIME/WordDecoder.pm
/root/.cpan/build/MIME-tools-5.411/blib/lib/MIME/WordDecoder.pm
Thanks

Josh
Reply With Quote
  #501  
Old 10-21-2002, 11:10 PM
NgtCrwlr's Avatar
NgtCrwlr NgtCrwlr is offline
 
Join Date: Jan 2002
Location: Fort Myers, FL, USA
Posts: 46
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Josh, try adding the following line to your newnews.pl

use lib "/usr/local/lib/perl5/site_perl/5.8.0";

Add the line right under the first line like this...

Quote:
#!/usr/bin/perl
use lib "/usr/local/lib/perl5/site_perl/5.8.0";
Not sure it will work, but it's worth a try

Cheers!
Reply With Quote
Reply


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 03:10 AM.


Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2024, vBulletin Solutions Inc.
X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.09245 seconds
  • Memory Usage 2,328KB
  • Queries Executed 25 (?)
More Information
Template Usage:
  • (1)SHOWTHREAD
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)ad_showthread_beforeqr
  • (8)bbcode_quote
  • (1)footer
  • (1)forumjump
  • (1)forumrules
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (1)modsystem_post
  • (1)navbar
  • (6)navbar_link
  • (120)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (4)pagenav_pagelink
  • (1)pagenav_pagelinkrel
  • (11)post_thanks_box
  • (11)post_thanks_button
  • (1)post_thanks_javascript
  • (1)post_thanks_navbar_search
  • (11)post_thanks_postbit_info
  • (10)postbit
  • (11)postbit_onlinestatus
  • (11)postbit_wrapper
  • (1)spacer_close
  • (1)spacer_open
  • (1)tagbit_wrapper 

Phrase Groups Available:
  • global
  • inlinemod
  • postbit
  • posting
  • reputationlevel
  • showthread
Included Files:
  • ./showthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/functions_bigthree.php
  • ./includes/class_postbit.php
  • ./includes/class_bbcode.php
  • ./includes/functions_reputation.php
  • ./includes/functions_post_thanks.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • showthread_start
  • showthread_getinfo
  • forumjump
  • showthread_post_start
  • showthread_query_postids
  • showthread_query
  • bbcode_fetch_tags
  • bbcode_create
  • showthread_postbit_create
  • postbit_factory
  • postbit_display_start
  • post_thanks_function_post_thanks_off_start
  • post_thanks_function_post_thanks_off_end
  • post_thanks_function_fetch_thanks_start
  • post_thanks_function_fetch_thanks_end
  • post_thanks_function_thanked_already_start
  • post_thanks_function_thanked_already_end
  • fetch_musername
  • postbit_imicons
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • postbit_display_complete
  • post_thanks_function_can_thank_this_post_start
  • pagenav_page
  • pagenav_complete
  • tag_fetchbit_complete
  • forumrules
  • navbits
  • navbits_complete
  • showthread_complete