fastforward, i would recommend a search at deja to try and find out which newsgroups it was in, we may be able to narrow it down for you.

ok it only came up with 4:

http://www.deja.com/dnquery.xp?QRY=b...ncurrent&DBS=2

they must have deleted the rest?

Thanks Chris

Those are all good posts. Well the non-test ones are anyway. There was also a bunch more I posted to 01alt.test, but they should be ok. I mean that's what test groups are for right? Surely they couldn't be construed as spam.

i seriously doubt it, what did hostpro actually tell you? did they say which newsgroup or what? if they didn't then ask them, it could very well be some competing website that just sent them an email saying you where. I bet they didn't even check it themselves.

Unfortunatly because we all hate spam so much some people are too quick to blame people who do nothing wrong. It dosen't appear anyone else would have done it i think most are waiting for vb 2.0 before they play with this so i would think it is just someone being a pain.

p.s. you using http://www.newsfeeds.com - not happy about them, their article numbers on the front are their goliath server, huge huge difference to their other spamkillers server which has massivly less posts in it. Probably why we end up with so many in the temp table when really there should not be that many (how many news threads go on for more than a month say and they should have a months worth of posts). Have to cancel my account to re-sign up on the goliath server. akk...

Yeah, I did ask hostpro. But the girl who sent the email has left for the weekend. The only thing in their notes was something about spamming newsgroups. No details. I just realized though; the organization field also contains my site address. There's no way to search for that though. I guess I'll just have to wait until Monday.

As for the spamkiller server problem. I did try goliath, but it was way too slow. I even tried the one reserved for platinum account holders. Spamkiller was by far the fastest.

Anyway, thanks for your help.

By the way, I've made a lot of changes to the script. It now italicizes quotes from usenet messages (we'll have to wait for v2 for colours. I also fixed a threading problem when a local reply is made to a locally started usenet thread. The problem was the message id does not exist until the message reaches usenet. I added some logic to update the local message with the message id on the next pull. Unfortunately I had to add some more fields to the tables and also renamed a few fields. You'd have to reinstall and re-pull history if you wanted it. Or you can wait for the version 2 release. The code is a lot tidier now. Let me know if you want a copy.

regards

sounds very cool, i will wait until v2.0 to try again, i was really just trying it out and playing around with it first and it is going to be really great feature for my site i know!

on the whole newserver thing, they have a new on called goliath2 which they say is a lot faster, it also has more files in it as well, did you every try that one? i am uite keen on getting as many threads to start with as possible.

otherwise i wonder what other companies are good. One thing i like about this one is that they don't actually have your credit card details but use that external comapny. This means if they do try and do their trumped up $50/hour spam removal charges if one of our users does something silly or whatever then they can't actually charge me, something i am a bit worried about.

anyway, looking forward to the v2.0 release of this :)

I really want to try this on my board. but I need to ask a couple things firsrt.

Where exactly would the newfeeed come from, would I have to pay for this or can I get it for free.

When installing the cpan modules, could I damage my server at all??

And lastly, the d/l link wont work, can someone email it to me? Cheers :) <G>

newfeeds.com you pay for. You can try to find free ones but good luck in your search :) (search for free usenet at dmoz.com for sites on it).

downloading cpan modules, i doubt it but don't know. I have never had any problem before however with these i think it tried to install perl again somewhere, i cancelled it and nothing appeared to go wrong although i probably have a few pointless files somewhere. That probably does not inspire you with confidance but i have installed quite a few modules this way and never ever had a problem and this was probably me just not understand what was going on so i wouldn't worry. Otherwise just use the modules in the file.

Can't send it to you, it has probably been taken down to sort the headers and things out.

I've never had problems with CPAN. The attempted re-install of Perl that chris is talking about was probably while installing netlib. If it finds it is already installed it will ask a whole bunch of questions regaring smtp servers and host names etc. Most of the modules will most likely be already installed if it is a managed server. The modules already on HostPros vservers were

DBI
DBD:MySQL
Date::Parse (part of the DateTime package)
Net::NNTP (part of the netlib package)
Net::SMTP (part of the netlib package)

So I only had to install one module:
Text::Autoformat

By the way. The download link is broken while I had email notification for replies and italicized quotes. There's a few other fixes too. I'm modularizing it so it will be easy to just drop in the section for vb2 without redownloading and reloading all the posts.

regards

fastfoward,

any plans for the email notication idea:

basically, someone in usenet starts a thread, member a comes along to forums and posts, a usenet member then posts a reply which gets uploaded to the forum. I then want it to notify agent a.

how?

well, i guess it could work like.

while it moves posts from the temp to the actual post section, it does a search for all the members who posted in that thread who have email notification on and sends them the email.

The only problem i can see with that approach is if you get two posts from usenet to the same thread, how to stop it sending two emails.

perhaps you have a better idea?

I'm adding that feature right now. It will work exactly as you suggest. It will only send an email once for every usenet batch pull. If you get 3 messages in the thread during a batch, only one mail will be sent to each user. Then if the next batch contains a message, it will send another email.

I'm trying to figure out how to use the existing vb template for email from within the perl script at the moment.

I'm also adding a feature to allow vb style quotes for usenet messages. It will mean you can choose italics or the <QUOTE> method. The choice will really depend on the newsgroup and whether the majority of posters use sensible news programs that apply quotations correctly.

regards

i would imagine this is something that is going to change a bit with vb2 with the whole vbhome type thing so hope not too hard to change over, glad you are implementing this though :) cool.

i don't think the whole (quote) and (/quote) thing will work too well, the problem is often that the lines of text get cut off so you get something like

> sdgfasdfgksdgfjdsgkdskgfjsdgjdkg nmfg dsfn mcnv bmcxvn bmxcvnb
df
> sdgfdsfgsdkjgfdksgf kdjfg kdfg dfn gmdfng dng mdfng mdfgn
dsdf
> skf skdg dmfg dkg kdfg dfgdf,gnsdfmgdsfgmsdgf
>

which really does not look good.

I just wonder if there is any clever logic that could be used more than the > line to take into these situations into account? i will have a think about it.

--

it is also the one time that i wish that vb allowed threaded listings like wwwthreads :)

oh the other thing was being able to use more than one usenet server.

The way i could see this being done is having a table containing usenet information,

ID
server address
username
password

etc.

and then for each newsgroup you have an extra column for server id. something like that.

if possible i would really like it, there is this newsgroup i really want to include but it is only avaliable on one server.

any help on where I can obtain a news feed from, cheap as possible please :)

My favourites are:

OnlyNews for $12.95 with as they don't throttle download speeds.

or

Newfeeds as they have a whole bunch of different servers to choose from.

but

If you're talking about a traditional IHave feed for your own news server then you are looking at a minimum of $200/mth.

Due to popular request... well 2 requests, I have cleaned up and released a new version of the usenet gateway script.

Some of the new features are:

• email notification to usenet replies
• emoticon translation into vb icons
• hyperlinked urls in messages
• italicized quotes
• support for xrover
• logging of outgoing posts

In addition, the code has been completely re-written and is much cleaner (at least I think so).

Some things planned for the version 2 release

• support for multiple news servers
• canceling of messages after they have been sent to usenet
• treating usenet posters as special members so that posts are counted and can be searched by username
• control panel integration

The url for the latest version is usenet_gateway_v115.tar.gz

regards
PAJ

Will this usenet gateway be included in vb2.0?

No, but this hack is being ported over.

cool fastforward, it is looking really great. I am looking forward to the vb2.0 version ;)

This is a very cool hack.

Though I wonder how full the database might get with a busy newsgroup.

Have you tried it with a busy one?

And, how does it handle any binary attachments?

I think it doesn't support MIME. It would be cool tho :)

This is a very cool hack.

Though I wonder how full the database might get with a busy newsgroup.

Have you tried it with a busy one?

And, how does it handle any binary attachments?

Binary attachments are not supported. Although it would not be difficult to add a feature that saves the binary attachment to a directory and post a link in the forum.

There is a variable called $max_msg_length that can be set to the max number of lines that a message can be before it is assumed the post is a binary one. The default setting is 300 lines. In addition there is a spam filter to check for 'begin 644' in the body and reject the post if it is found (begin 644 means the post is binary).

As for busy groups. It all depends on your news server speed, your connection speed and the amount of space you have available for storage. But for practical use where you would maybe have 10-20 newsgroups, there will be no problem at all.

There was bug in the expire_article routine that was causing articles never to be removed from the usenet_ref and usenet_article tables. The fix is only in the newnews.pl script and affects only one line.

The link in the first post of this will always point to the latest release.

Hi...me again. I ran the SQL changes, no problems there (although the spam table isn't showing up on the database...any idea why? Is this required?)

Anyways, made the modifications to the newnews.pl file, uploaded to the cgi-bin directory (I don't believe my web host will allow to execute scripts outside of CGI-BIN), chmodded it to 755. Now I'm getting an Internal server error when I run the script from my browser.

I updated the first line of the script to my local path to perl (I know this works because I had UBB up and running before vbulletin, and I just copied that from the cgi script that was working). The only question in my mind is the following line in newnews.pl:

#use lib "/[path.to.your.local.perl.directory]/perl5/site_perl/5.005";

What is the proper syntax for this line? Could this be causing the internal server 500 error I'm seeing? My path to perl is #!/usr/bin/perl

Another question...why are there two newnews.pl files in the distribution? Which should I be using? The one in the Rcs folder looks different than the other one (header information) so I assumed that wasn't the correct one.

Thanks in advance!

The usenet_spam_filter table IS required. The script will fail if this is not here. It may be the cause of your error. If this is not being created, there may be an error in my SQL. Try running just the part that builds this table.

You don't need the '#use lib' line as your ISP has installed the modules for you.

The shebang line is probably correct. If you had ubb running using that line then it definately is.

The RCS sub directory is for revision control. You can ignore it. I just included it for completeness.

One point though, I did not intend this script to be executed via the browser and I have no idea whether it will work or what implications it may have. I will do some testing over the next few days and let you know the outcome.

regards

Not sure why the spam table wasn't added, but I ran just that part of your script and it did add the table to the database.

I'm kinda baffled at this one...I changed the newnews.pl again, re-uploaded it to my cgi-bin directory, Telnet'ed into my account, and I can't find the file there!!!! It shows on my FTP program, and I can even hit it with my browser (although I get that same 500 error), but looking for it through the telnet shell, I can't find it! I want to execute it from there so I can see if/what the error is, but can't find it.

Yes, I did CHMOD it to 755, even tried 777 and it just doesn't show up. Any thoughts?!?

stupid telnet program...found the program, so I've now run newnews.pl via telnet. Connects to the server and the right newsgroup, grabs the first 100 messages, when it does to dump them to the database, I'm getting the following error:

Requested 100 messages... 0 not available or rejected.
DBD::mysql::db do failed: You have an error in your SQL syntax near ''No User',9
79507486,'ALISHA has done the following\n____________________________' at line 1
at ./newnews.pl line 419.
Query failed (INSERT IGNORE INTO post (allowsmilie,threadid,username,dateline,pa
getext,visible,msgid,ord) VALUES ('1',,'No User',979507486,'ALISHA has done the
following\n_______________________________________ _\nSlept: 7 Touched: 31750 Dro
pped: 37\n________________________________________\nNetb ios Rulez \\GUYC \\C \\E
SC400 \\C \\ACI \\BOB PUBLIC \\C GATEWAY \\CUSTOMERS \\DOWNLOADS \\MY DOCUMENTS\
n\\MYLABEL \\PROGRAM FILE \\COMPAQ5695 \\MY MUSIC \\MYDOCS \\C \\PAVILLION A \\P
AVILLION C \\backup \\D \\C \\D\n\\E \\C_PAZR1 \\HOME C DRIVE \\TEMP \\C \\C \\C
\\C \\C \\D \\Downloads \\SHARED \\HHC \\HHPROGRAMS \\HHSHARED\n\n..and the wor
ms ate into his brain\n','1','<0e8cc39d915d766e63245db40b0119cb@an on.xg.nu>','0'
)) at ./newnews.pl line 419.

Any ideas here? I'm at a total loss here. It looks like it's having a problem dumping the messages into the post table, but I'm not an SQL expert).

On another note, a couple of things you might want to consider when you do your next documentation update:

1.) The SQL changes (textfile) - There are two usenet_outgoin_log table create commands

2.) You are using the table POST and post interchangably, which is causing errors in the alter table section toward the bottom of the query. Took me awhile to figure that mysql is case sensitive.

3.) You might want to make some recommendations about a mysql client. I'm using phpMyAdmin, and it's great. I couldn't imagine trying to do this stuff manually.

That's it for now! Any suggestions on the above?

The problem seems to be that 'null' is being inserted into the threadid column. As far as I can see, there are two possible reasons for this error:

Number 1 is line 292 that reads
is failing for some reason. mysql_insertid is a built in mysql function and returns the id of the last inserted record. I have no idea why this would not work on your setup. What version of Perl and MySQL are you using?

Number 2 is if the query on line 306 was pulling back null for the threadid column from the thread table. Now this is virtually impossible as threadid is an auto_increment column with a unique index on it.

I don't want to ask the obvious, but I will :) you are installing this on version 1.15 or 1.114 right?

I'll need to do some more investigation. I'll get back to you.

As for the SQL instructions. I'm an Oracle DBA so I'm used to tablenames only being case sensitive if you put double quotes around them. :) I will correct them asap.

regards

Not a dumb question at all. I am running:

vb 1.1.5
mysql 3.23.32
perl version, well, as you know, hard to determine, but if I issue the command $perl -MDate::Calc -e 'print "$Date::Calc::VERSION \n"' the version returned is 4.2.

Anything else? I've tried a bunch of different stuff, manually wiped out the records in the usenet_article and usenet_ref tables, reinstalled everything (including vb), with no avail.

Thank you, looking very forward to getting this up and running, and your support is better than most technical support companies!!!!!!!!!!!!!!!!!

-RC

OK... looks like you have uncovered a little buggette. I think the problem is this.

When a usenet article is received that will become the thread starter, an entry is made in the thread table. The statement that does this is on line 264 and uses the INSERT IGNORE clause to prevent any possibility of duplicate messages if your news server decides to reindex and change message numbers on you.

However, the side effect to this is should the statement ever actually be ignored due to a duplicate messageid, the next line that gets the mysql_insertid will return null as nothing was actually inserted. So consequently, when it tries to put the corresponding entry into the post table on line 266, it will fail because threadid in the post table is specified as NOT NULL.

Now this is all very well and is easy to fix, but the real issue is why you came across a duplicate. The message id column is supposedly unique across all of usenet. You mentioned you tried again by emptying out the usenet_article table first. If so, the only way a duplicate could occur would be if the article came twice with the same messageid from your news server. But even this shouldn't matter as the usenet_article table has a unique index on the 'msgid' column.

I'll fix the script and post it in about an hour. In the meantime, check the following:

• make sure the only unique keys on the thread table are 'threadid' and 'msgid'
• make sure the usenet_article table has a unique index on 'msgid'

regards
PAJ

Well, glad (and sorry) to hear it doesn't look like it was me. Sorry to put you through this, but I guess that's the nature of the beast, huh? ;)

Unique keys exist as you state in your message above. Really wierd I agree. At any rate, look forward to your reply.

Thanks again!


---------------------------------
Captnroger
Forum Moderator
Gofishohio.com
www.gofishohio.com

The fix has been incorporated into the script. The newnews.pl and the sql instructions are affected.

The change simply checks whether the $threadid is null prior to inserting into the post table.

If this doesn't work for you captnroger we'll need to review the entire setup.

If your problem was actually caused by a faulty mysql_id function you will not see any messages in your forums.

PAJ
--
the latest version of this script is always in the first post of this thread.

Here's the error I'm getting now (I did run the SQL query without an error though!) Modified the new newnews.pl, ran it, here's the message:

DBD::mysql::st execute failed: You have an error in your SQL syntax near '' at l
ine 1 at ./newnews.pl line 424.
Query failed (SELECT title FROM thread WHERE threadid=) at ./newnews.pl line 42
4.

Sorry to be a pain. Any thoughts?

OK.. assuming the problem is due to duplicate messageids the problem was my fault. It was trying to index that non-existent thread. I've added another check to stop it happening.

I still may have missed some so let me know if it fails anywhere else.

regards

FF -

First of all, I commend your efforts today to help me get this working. You truely are a champ.

The latest mods to the script got me through the importing process, YEAH! The script executed without an error.

What DID happen though is that while my forum has all the headers, along with the author, date and time, if you click on the actual post, there is nothing there. In looking at the database, nothing is actually being posted to the post table. I have three test messages in there that I created, that's it. The newsgroup messages don't make it to the post table. Looks like they are still in usenet_article.

So close!!!!!!!!!!

That indicates that something really is wrong with the 'mysql_insertid' function call or for some reason it's returning null.

Remove the IGNORE word from the SQL in line 303 so that it reads:
and add the following debug line right after line 304 that reads $threadid = $dbh->{'mysql_insertid'};
Now if the SQL doesn't fail we know it is the 'mysql_insertid' function at fault. If the SQL fails then show me the error message as you did before.

The other thought I have is maybe your news server doesn't fully support the 'xhdr' function. This is unlikely but it would cause any header it couldn't find to be null.

What provider are you using? To check if this the problem I can give you temporary access to a test DNEWS server I have at home. It will be slow though as I only have 256K upload speed. Let me know your IP and I'll add access for testing.

regards.

hmmm....interesting. Never considered it may be the newsfeed. I'm using newsfeed.com, although I am using their spam-free newsfeed. Let me play around with this a bit...I'll get back with you.

okay, just to be safe I wiped out the database and started over again, and also changed news feeds. Here is the debug of the session as per your request:

Connecting to News.newsfeeds.com... Connected
Sending authentication info... Authenticated and logged in
Getting article batch from alt.test
Fetching headers of articles 848576 to 848675... done
Fetching article body 848576... OK
Fetching article body 848577... OK

much..
deleted..
here..

Processing article batch...
Requested 100 messages... 14 not available or rejected.

DEBUG: threadid for msgid '<20010207001313.30401.qmail@nym.alias.net>' is
DEBUG: threadid for msgid '<7daeb90ada844104ea268ca9b8bc25dd@mix2.hyperreal. pl>'
is
DEBUG: threadid for msgid '<mm0g6.140$Co5.19947@nntp2.onemain.com>' is
DEBUG: threadid for msgid '<8270e04f916966896207f88be635ef2b@anon.xg.nu>' is
DEBUG: threadid for msgid '<20010207000811.27948.qmail@nym.alias.net>' is
DEBUG: threadid for msgid '<Ek0g6.225629$w35.39920301@news1.rdc1.nj.home.com >' i
s
DEBUG: threadid for msgid '<Nk0g6.1085$D3.4198@tor-nn1.netcom.ca>' is
DEBUG: threadid for msgid '<ih0g6.129$0W5.38111@nntp3.onemain.com>' is
DEBUG: threadid for msgid '<Cc0g6.124$0W5.35089@nntp3.onemain.com>' is
DEBUG: threadid for msgid '<95q1pn$4id$1@lacerta.tiscalinet.it>' is
DEBUG: threadid for msgid '<JW%f6.118$0W5.21387@nntp3.onemain.com>' is
DEBUG: threadid for msgid '<S_%f6.127$Co5.9928@nntp2.onemain.com>' is
DEBUG: threadid for msgid '<A30g6.129$Co5.13100@nntp2.onemain.com>' is
DEBUG: threadid for msgid '<95q2nk$hv28s$1@ID-55073.news.dfncis.de>' is
DEBUG: threadid for msgid '<d80g6.131$Co5.13778@nntp2.onemain.com>' is
DEBUG: threadid for msgid '<3a808f06.949583@news.attcanada.ca>' is
DEBUG: threadid for msgid '<uR%f6.122$Co5.7773@nntp2.onemain.com>' is
DEBUG: threadid for msgid '<nM%f6.111$0W5.13742@nntp3.onemain.com>' is


Lots of these.... ends with...
DEBUG: threadid for msgid '<qk2g6.929$0W5.133222@nntp3.onemain.com>' is
DEBUG: threadid for msgid '<3p2g6.932$0W5.135574@nntp3.onemain.com>' is
DEBUG: threadid for msgid '<Bv2g6.395$Co5.72310@nntp2.onemain.com>' is
Processing outgoing messages


Does this help? I doubt it's the feed (or maybe it is, I dunno) because I'm using newsfeeds.com.