Usenet gateway

[Dontom]

Hi together,
I installed the script and everything works fine, my users love it -
now some of the usenet - users told me they will sue me if I copy their post to my forum!
I thought usenet was public???
Is there any chance that I can use X-NoArchive: Yes or exlude by posters mail adress?

BTW, I use the gateway read only to avoid problems with the newsgroup - I just don't get it...
The newsgroup deals with cycling, the other with mountainbiking...

a shocked Tom

[fastforward]

Originally posted by Dontom
Hi together,
I installed the script and everything works fine, my users love it -
now some of the usenet - users told me they will sue me if I copy their post to my forum!
I thought usenet was public???
Is there any chance that I can use X-NoArchive: Yes or exlude by posters mail adress?

BTW, I use the gateway read only to avoid problems with the newsgroup - I just don't get it...
The newsgroup deals with cycling, the other with mountainbiking...

a shocked Tom
Once they submitted their post to usenet they submitted it to the public domain. Threaten to counter sue for wasting your time :)
You think Google and the bazillion other usenet/mailing list archives are paying royalties to everyone?

It is could manners to comply with the x-noarchive flag though (even though this isn't required as part of the usenet RFC). I'll add the option to the script.

[chrispadfield]

bringing back the usenet discussion.

At the moment i am using spamkillers.usenet.com

The main reason is it seems to work ok but secondly the company does NOT have my credit card. Some credit card intermediary does, ibill i think and having dealt with these people for my script they do everything and i mean everything in the consumer's favour. If you emailed them after using your service for probably 2 months and said "i don't like this, it is not what i expected, i want a refund" they would give you one. No kidding.

What is the point, the point is that if something ***did*** go wrong there is nothing that this company could do about it. They couldn't just charge you some crazy hourly charge for their clean up operation. Of course this should not stop you making it happen but it does reduce your liability. The only thing they can really do is email your isp/webhost provider about it. They are likely to be more understanding.

Despite this we are of course all breaking the TOS. If anyone wants to poll a commercial feed then i am definitely up to contribute. If we can get one for $100 a month and 10 of us pay $10 we are doing well. Our bandwidth is going to be insignificant with most people having less than 25 newsgroups, i would imagine on average that is what, 10MB a day absolute maximum. It would require one person to sign up and the rest of us to send in $10 a month. (although whether you are allowed to resale is another point!).

[chrispadfield]

I am so nearly there :)

last thing. What would be the code i need to get this to run via cron but to create a log file of the output of the shell script (so i can check it is running properly).

i can work out the cron time bit but do i just add > /path/to/log.txt sort of thing?

[chrispadfield]

also, do you think you could release some standard spam filters?

[fastforward]

Originally posted by chrispadfield
I am so nearly there :)

last thing. What would be the code i need to get this to run via cron but to create a log file of the output of the shell script (so i can check it is running properly).

i can work out the cron time bit but do i just add > /path/to/log.txt sort of thing?
To generate a log from your cron job use something like this:

0 0,4,8,12,16,20 * * * nice -20 /home/dbforums/newnews.pl >/var/log/misc/newnews.log 2>&1

If you want the output emailed to you, just leave the redirection bits off completely. The output will be mailed to the owner of the cron job.

[NORRITT]

where can i download the newest version of this hack für vB 2.0 final?

[Gilby]

Originally posted by NORRITT
where can i download the newest version of this hack f?r vB 2.0 final?
The latest version is always in the first post of this thread, and also at http://britishexpats.com/download/usenet_gateway.tar.gz

[NORRITT]

ok, thanks, just thought, that this version is for the beta, not for the final

[Dontom]

I'll add the option to the script.
Can you already estimate when a next version will be released?
Thanks in advance

[fastforward]

Originally posted by Dontom

Can you already estimate when a next version will be released?
Thanks in advance
I've been really short of free time lately but I hope to get a new version out this weekend.

[antigravityboy]

Any hope that binary attachments will be in the new update. I am very excited about being able to add this feature to the bulletin board.
:D

[fastforward]

Originally posted by antigravityboy
Any hope that binary attachments will be in the new update. I am very excited about being able to add this feature to the bulletin board.
:D
I plan on having a very early test of it available. It will be an option in the CP. If you turn it on, I wouldn't risk using it on a production site just yet. :)

[antigravityboy]

Glad to hear you are already adding the option. I can imagine how it will have to be beta-runned with the alt.binaries groups though. Some of the groups have a ton of pictures and that would fill up the database really fast.
Thanks again for adding the option:D

[NgtCrwlr]

I'm setting up all usenet forums as moderated posting only so I can keep a close eye on the spam issue.

I've found that if I set a forum to moderated posting, newnews.pl does not check the moderator validated flag and will post to usenet even though the post is still pending validation.

Also, I am having problems with local reply posts not making it to usenet. I'm wondering if this could be related to the moderated posting only setting. I didn't have this problem when I was testing without moderated posting on previous version.

The threads MIS Test 1 and MIS Test 2 show local replies but the replies do not show on usenet, and MIS Test 3 is a post waiting validation and dose not show locally, but does show up on usenet.

http://www.mis-forums.com/vbbs/forumdisplay.php?forumid=67

news:alt.test.test

I hope you're making progress on the x-no-archive switch. That switch and moderated posting are the last pieces I need before opening the doors to usenet posting on our forums.

[Sinema]

Howdy,

I'm having a problem .. I can't figure out what it's supposed to do

I've installed it exactly like the instructions said..

I've added a newsgroup .. and directed to a forum .. but nothing

it says in the Use Net Options 61 posts . then I goto the main forum and nothing is there..

[NgtCrwlr]

Hi Sinema,

A link to your BBS may help in figuring this out.

Does the forum that you setup for your usenet posts show up empty, or not at all? Double check all of that forums CP settings and make sure that you have forum active checked yes.

That's about all I can come up with right now.

[Sinema]

<a href="http://kickme.to/buffybb/" target="_blank">http://kickme.to/buffybb/</a>

See it doesn't do anything...

I've created a forum called

alt.tv.buffy-v-slayer

and directed that newsgroup to go there..

but the forum you can see .. anything else is gone.

and I have no errors anywhere

I'm hoping I can get a set up like dBforums.com has .. now that is awesome :-)

except of course mine would be the alt.tv areas lol but that place is nice!

[Sinema]

Also I'm curious

Do I put my ISP info into the username and password slot to log into a news server?

I placed my ISP's news server in the default slot because I had no idea what to put there..

[NgtCrwlr]

After you installed this hack have you been able to display *any* vBulletin pages? If not you have to go back and figure out why... somewhere along the line you made a mistake that is not allowing vBulletin to work.

Even without posts in your usenet forum you should still be able to reach other forums and it looks like you can't so this is a more sever error you will have to backtrack to find.

For the main news server you can try your ISP and if your ISP requires login and password you must supply that too. The news server and password fields under the individual groups are for times when you want or need to pull news from different servers for specific groups that may not be available on your main news server. They can be left blank until you have a need.

Hope this helps... I'm out for the day, I need some sleep. Good luck!

Cheers!

[Sinema]

I installed it exactly how their instructions said and the instructions said that if there were any errors I would see them "loudly"

people can post in the forum , add threads etc.. just nothing from usenet is shown in the folder

[NgtCrwlr]

I forgot to mention the link you posted goes to a blank page so I couldn't see anything.

If you can run newnews.pl from telnet without error and it shows a clean disconnect message from your news server then I'm guessing that you have your forum settings wrong.

Try posting a link that works and we can work from there.

[Sinema]

The cgi-bin server of the place I use is down ..

and also

since it's a free web host .. that allows php and mysql ... I do not have telnet access

I followed their directions exactly as they listed.. and I didn't see any huge errors or anything.. I actually did not see an error at all ..

[Gilby]

Originally posted by Sinema
I followed their directions exactly as they listed.. and I didn't see any huge errors or anything.. I actually did not see an error at all ..

Did you run newnews.pl? This is done from telnet... although, the latest version may be able to be ran from the webbrowser. Can you show us the output of what you get when you run the newnews.pl script?

[Sinema]

Since it's a free webhost ..

The only web host that I know of that is free and supports mysql/php

It doesn't allow telnet connection :-(

so I can't telnet to the script but the instructions didn't say anything about the pl

it said to place it above the wwwroot .. so I placed it right above www in my ftp

[Gilby]

Originally posted by Sinema
it said to place it above the wwwroot .. so I placed it right above www in my ftp

That script is meant to be run as a cronjob and is the script that fetches the postings from the newsgroup. So without running that script, no posts will go to or from the newsgroup.

Since you don't have telnet access, you'll need to put the newnews.pl script in a place where it is visible from the web. Also you will need to add this line near the top of the script:

print "Content-type: text/html\n\n<pre>";

Then run it from your web browser. You'll have to figure out a way to run it periodically too since you don't have access to a cronjob. Maybe there is a service on the internet that will access a certain page every 15 minutes or so and therefore execute the script.

[fastforward]

The newnews.pl was never intended to be executed from the browser. As per section 2 of the instructions:

Upload the newnews.pl script into any directory and chmod 0755 to make it executable. It is recommended that this directory be outside your wwwroot. This script is not designed to be executed as a cgi. It should be executed periodically via cron. Next, upload the usenet.php file into the admin directory of your vB installation. This is the control panel page.

And then towards the bottom under the heading 'Running':

To collect all the historical news on the server you will need to set the 'keep orphan replies' option to an artificially high number to prevent losing posts. Once you have collected all the news, reset the number to somewhere between 2 and 5 days.

Don't worry if when you run it the first few times you see a lot of messages on the screen saying message 'Not Available'. This is because most newservers are inaccurate in their reporting of the first and last article they have available. Just keep running it and you will see articles start to collect eventually. This is only an issue during the initial historical pull.

Once you have the history, you can set up a cron job to run the script every 20 minutes or so.

But that aside... most ISP news accounts authenticate you using an IP address (the IP address of your home PC). Unless the web host has a news server you can use, you will probably need to get a seperate feed.

As for running the newnews.pl from the browser; I believe it has been done successfully, but I have not tested newer versions using this method and make no guarantees that there won't be problems.

NOTE to everyone waiting for the next release:
I've been so so so so busy :) I haven't made much progress with the binary attachments so the up-coming version will only have the x-noarchive addition plus a few bug fixes. I was able to reproduce the quote color problems. It was a syntax problem that sonehow crept in between versions. Probably a slip on the keyboard :)

[Sinema]

Wow thank you I didn't even know that

And good point ..

I know where there is a server that allows Telnet and CGI .. etc..

but it doesn't run php or mysql ..

do you think I can use that for anything?

[Gilby]

Originally posted by Sinema
do you think I can use that for anything?

If you can make a cronjob from that account via telnet, then you can run the following command with your crontab file:
wget http://domain.to/your/newnews.pl --output-file=/dev/null

[chrispadfield]

Originally posted by Sinema
Wow thank you I didn't even know that

And good point ..

I know where there is a server that allows Telnet and CGI .. etc..

but it doesn't run php or mysql ..

do you think I can use that for anything?

yes, just run the script on that site and set the database to the one on your other host. That is what is so good about databases, you don't need to run the programs at the same place.

[Mark Hewitt]

I wonder how many people are actually interested in binary posts.

I wouldn't want you to be wasting your time on a feature nobody is interested in?

[v0n]

Well.. with the binaries function Usenet "hack" (no longer should be called that - it's the Whole Development already, *me bows to fastfoward* ) would be an ultimate and scalable news board. With a proper server(s) and maintenance you could create anything from a small hobbyst website to commercial usenet portals.

However, before it gets to that stage, I for example still have problems with newnews.pl - posts never inserted into threads or never retrieved from news servers, posts disapearing when manualy inserted from admin menu to correct threads (they just never appear there) etc..

[robertusss]

we should get an own forum for this hack... 36pages... phew...

I lost the discussion on the newsfeeds - can anybody here recommend a cheap service?

[Mark Hewitt]

Originally posted by robertusss
[B]we should get an own forum for this hack... 3

Agreed. It's a pain having to discuss it all in one thread :(

[NgtCrwlr]

Originally posted by robertusss
we should get an own forum for this hack... 36pages... phew...

I lost the discussion on the newsfeeds - can anybody here recommend a cheap service?
I've been quiet about what I've found on the news feed provider situation because I haven't been able to really test posting yet. I didn't want to get everyone excited and then find out there were a bag full of problems down the road.

Anyway what the heck... you're all big boys and can decide for your selves right? Here's what I came up with... several pages back I started to get paranoid about breaking news server TOS agreements and that led to too many hours sifting through all the crap out there. BUT, I did find what I think may just be the golden needle of the haystack.

I sent Newscene.com several e-mails and received very promising and prompt responses. I signed up for three months at $36.99 and everything seems to be going just great. Aside from the fact that I have not tested posting other than alt.test.test.

The great deal about these guys is they are fully away of what we are doing pull/post accounts and said they have no problem with this. Here's a couple snips:


> I noticed the following in your TOS
> 12. 'Sucking' feeds are prohibited unless authorized. 'Sucking feeds
> will require an additional charge. 'Sucking feeds' will not be provided
> to individual users. Newscene reserves the right to terminate an account
> used for an unauthorized 'sucking feed'.

The "sucking feed" prohibition is mostly a leftover from the days when we offered flat-rate service. It does still apply in some situations--not yours, though, so you can set up with any type of account we offer (see our Pricing page for details). If you expect to pull down only 10 Megabytes per day, then the Standard account (which allows 5 Gigabytes per month) should be more than adequate. Please let us know if you have any other questions or concerns.

> I need to know that Newscene will be understanding in spam issues and
> treat us as a provider and not as an individual, and be willing to work
> with us to solve these problems. Rest assured, we plan to attack spam
> with a vengeance, and we will be willing to do whatever we can to avoid
> problems.

Yes, we're willing to work with you, but because of the way pull/post-style software works, this will be a bit tricky. All of the messages posted at your site will come through tagged with your username when they are posted to our servers. We automatically filter spam at the user level, and if our spam filter is tripped for your account, all of the articles posted from your account will be filtered until we figure out what's going on.
This means there is a possibility that legitimate posts (which are sent in the wake of a wave of spam) may be dropped. If you can live with the shadow of that possibility, then I think everything will be fine.



I also sent http://www.newszilla.com/ the same e-mail questions and they are another candidate at a much higher price though
In the interest of keeping it simple, if you are running your own server and managing user authentication/posting there, we can allow one IP address (your server) to NNTP pull/post for $75 per month. This would be for the group set which you mentioned in your e-mail below. Let me know if we can be of service.
So there ya have it, I've been pulling from Newscene.com (30min cron job) for over a week without problem. I mentioned to the support guy that they may have a number of other vBulletin Gateway users signing up in the near future. They had no problem with that.

Pretty hard to beat at $12.99/mo and knowing that you are being up front with the company and won't be breaking any TOS agreements. http://www.newscene.com/ Add the quick response I got on the inital e-mails and it was a no brainer.

Cheers!

[NgtCrwlr]

Sheez, I guess I got a little carried away there... and forgot to address the FREE feeds. I had very good luck with pubnews.netcom.net.uk (slow but allows posting) and dp-news.maxwell.syr.edu (fast no posting).

http://pubnews.netcom.net.uk
You don't need auth/pw if you don't post, if you want to post you have to register... it's free though.

Cheers!

[Dioxin]

i have two users complaining, that they didn' get a reply on there newsgroup posts.

here is the posting on our forum:

http://www.aktienboard.com/vb/showthread.php?s=&threadid=5334

and here is a short overview about this newsgroup on newsgate:

http://www.newsgate.ch/de/de.talk.tagesgeschehen/index.html

check out the post from flinx 2000 (Re: Greenpeace erzieht hirnmanipulierte , Flinx2000). he didn't get an answer on our forum, but there exists answers in the newsgroup. :(

why doesn't the script fetch alle replies?? maybe it is my fault, and i have false options set in the cp....


Please, who can help me fix it? Thank you very much..

Dioxin

[v0n]

Check in
Control Panel --> Usenet --> Groups --> Review/Insert Orphans, you might find some posts sitting there and waiting for manual injection. In my case majority of replies to cross-posted threads get "confused" and end up there...

[Dioxin]

there are a lot of posts to insert.

i think this isn't the best solution. how can i avoid this or why are there so many threads awaiting manual post insertion?

[Dioxin]

has nobody an idea to fix my problem?

there are a lot of posts, that aren't fetched by the script and wait for manual insertion.... what's the error?

please help

[Gilby]

Originally posted by Dioxin
has nobody an idea to fix my problem?

there are a lot of posts, that aren't fetched by the script and wait for manual insertion.... what's the error?

please help

Posts that don't get inserted are not inserted because the references in the headers of the email do not match any of the message ids of existing posts in the forum. If your newsgroup is also a mailing list, as the one i use it on is, then this will occur more often (due to there being so many crappy mail readers that don't use the same headers). I also use this script on a private news server and those newsgroups usually never have a problem (unless the post before it was a binary post and therefore not inserted into the forum). Do these uninserted posts only occur on posts that are in reply to forum posts? If this is the case, this could be because you have changed your useragent setting in the control panel (but that shouldn't affect too many), or that your newserver that you are using is changing the headers and removing that useragent. If the nesgroup has a lot of binary posts, all posts in responce to that binary posts (and the rest of the thread branching from the binary post) will not be inserted into the forum.

[fastforward]

Originally posted by Gilby

If the nesgroup has a lot of binary posts, all posts in responce to that binary posts (and the rest of the thread branching from the binary post) will not be inserted into the forum.
This problem really needs to be fixed. But without risking the logical flow of a thread, I'm not sure how. Any suggestions?

[Gilby]

Three possibilities are:

Have the binary posts be inserted as attachments with the post.
Have a table that will keep track of the msgid of the thrown away binary posts and its corresponding thread that it was to go in (the subject too as some threads start with a binary post).
Have the first so many characters of a binary post be inserted and cut it off with a message saying that it was a binary post.


Ultimately, the first option is the best one... but that's also the hardest/longest one to implement. That can be combined with implementing html multipart posts as well.

The 2nd and 3rd options are easier to implement and can also be used as options for those that don't want to fill their database with binary files, so implementing that now would get the 'problem' fixed.

[v0n]

I have posting from the board disabled and get a lot of posts waiting for manual insertion. If I had to find a pattern I woud say majority of these posts were cross-posts to groups not present on my board (as in reply to message nr 11111 in rec.group.whatever, which I have and 22222 in alt.group.whatever which I don't). But (there's always but) many many times first and a number of cross-posted follow ups appear in vB and then next one just gets stopped. Some posts are kept on hold simply because first message in the thread wasn't fetched, although message is available on the server and if I reset the counter it will fetch it when forced.

Another bug would be mixup of posts from time to time - I had plenty of moments when full threads and replies to another group were inserted into different thread in different board. This is minor though.

Strangely enough everything worked perfect untill last versions for 2.0 betas (ones with all changes in the code)...
For the last five days I've been purgiung all posts from one of the groups and changing servers, suspecting there might be something weird about INN we have in the office, but pulling from private account and from public server also had the issues described above...

vBulletin 2.0.1 + Usenet hack 20010528

[Gilby]

Originally posted by v0n
Another bug would be mixup of posts from time to time - I had plenty of moments when full threads and replies to another group were inserted into different thread in different board. This is minor though.

I've had that happen... and i thought it might have been my database not being set correctly (probably sql not being executed from upgrade instructions). Some posts and the downline thread would show up in threads that only had the one post and was a forum post in another forum. Occasionally I would get the same post being inserted many times into the same thread (the correct thread). While the last problem may have been a one time thing, I fixed the other bug by making the database have a default value that was not null (in my case I used 'unicyclist') for the msgid field of the post. Since then, I haven't had any problems (that I've seen).

[shri]

Any ideas why I'd be getting the following error messages in the output?

DBD::mysql::db do failed: Duplicate entry 'reasonable' for key 2 at newnews.pl line 628.
DBD::mysql::db do failed: Duplicate entry 'interest' for key 2 at newnews.pl line 628.
DBD::mysql::db do failed: Duplicate entry 'rates' for key 2 at newnews.pl line 628.
DBD::mysql::db do failed: Duplicate entry 'amend' for key 2 at newnews.pl line 628.

Regards,
Shri

[markrt]

Hi

This look a great script on your Forum Fastforward.

I have couple of questions.

How do I execute the file newnews.pl via cron? would it be posible to set up a script so that when a member logs on, it runs the newnews.pl file?(I have also emailed my host this question).

vB code changes, (8) admin/functions.php.
You say to ADD the new code just before the closing brace of the indexpost() on or around line 1022,
My line 1022 is maked in red.
Where do I add your code? Is it at the green marked Add here????


if ($newwords) {
$newwords=trim($newwords);
$insertwords="(NULL,'".str_replace(" ","'),(NULL,'",addslashes($newwords))."')";
$DB_site->query("INSERT IGNORE INTO word (wordid,title) VALUES $insertwords");
$selectwords="title IN('".str_replace(" ","','",addslashes($newwords))."')";
$DB_site->query("INSERT IGNORE INTO searchindex (wordid,postid) SELECT DISTINCT wordid,$postid FROM word WHERE $selectwords");
}

if ($newtitlewords) {
$newtitlewords=trim($newtitlewords);
$insertwords="(NULL,'".str_replace(" ","'),(NULL,'",addslashes($newtitlewords))."')";
$DB_site->query("INSERT IGNORE INTO word (wordid,title) VALUES $insertwords");
$selectwords="title IN('".str_replace(" ","','",addslashes($newtitlewords))."')";
$DB_site->query("REPLACE INTO searchindex (wordid,postid,intitle) SELECT DISTINCT wordid,$postid,1 FROM word WHERE $selectwords");
Add here????
}
}

// ###################### Start makeforumjump #######################


Thank You for any help that you give.
Mark Thornton

[webhost]

Loaded this morning everything looks good but I have 1 probelm for sure maybe 2. The first is when I run newnews.pl I get this error.

Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, webmaster@realwebhost.net and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.


Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

Apache/1.3.19 Server at www.realwebhost.net Port 80

Second I went to control panel made a forum called test and it shows up. Then I went to groups for usernet, I added a new group and the only thing I changed was I pointed it the new forum test. After I save I go to test forum on main page but the only thing that is there is test forum It does not show default newsgroup underneath. Can someone tell me what I doing wrong please. I know it is probably simple.

Thanks Joey
HTTP://forums.realwebhost.net test forum at the bottom

[fastforward]

Originally posted by shri
Any ideas why I'd be getting the following error messages in the output?


DBD::mysql::db do failed: Duplicate entry 'reasonable' for key 2 at newnews.pl line 628.
DBD::mysql::db do failed: Duplicate entry 'interest' for key 2 at newnews.pl line 628.
DBD::mysql::db do failed: Duplicate entry 'rates' for key 2 at newnews.pl line 628.
DBD::mysql::db do failed: Duplicate entry 'amend' for key 2 at newnews.pl line 628.

Regards,
Shri
Looks like you have problems with your word table. This problem has been reported before by people not using the usenet hack. The SQL that runs makes it impossible to try to insert a a duplicate key so this implies the table is somehow corrupt.

Your best bet would be to re-index. If you have a large board and that's not an option, then either;

1) Simply delete those words from the word table.
2) turn off the index option for the news for a single pull.

[fastforward]

Originally posted by markrt
How do I execute the file newnews.pl via cron? would it be posible to set up a script so that when a member logs on, it runs the newnews.pl file?(I have also emailed my host this question).

Possibly. But you'll have to write some sort of hack for vB that calls newnews.pl when someone logs on.

vB code changes, (8) admin/functions.php.
You say to ADD the new code just before the closing brace of the indexpost() on or around line 1022,

Your code shows the addition being made before the closing brace of the if statement. It needs to be just before the closing brace of the indexpost() function. see below.

if ($newtitlewords) {
$newtitlewords=trim($newtitlewords);
$insertwords="(NULL,'".str_replace(" ","'),(NULL,'",addslashes($newtitlewords))."')";
$DB_site->query("INSERT IGNORE INTO word (wordid,title) VALUES $insertwords");
$selectwords="title IN('".str_replace(" ","','",addslashes($newtitlewords))."')";
$DB_site->query("REPLACE INTO searchindex (wordid,postid,intitle) SELECT DISTINCT wordid,$postid,1 FROM word WHERE $selectwords");
}
Add here????
}

[fastforward]

Originally posted by webhost
Loaded this morning everything looks good but I have 1 probelm for sure maybe 2. The first is when I run newnews.pl I get this error.

Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, webmaster@realwebhost.net and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

You're trying to run newnews.pl as a cgi. It should be executed priodically from the command line or with the help of cron.

It may be possible to run it as a cgi, but first, get it working from the command line.

[webhost]

Thanks for your quick reply, this is how I am trying to run it

http://forums.realwebhost.net/newnews.pl

Is this wrong?

Joey

[fastforward]

Originally posted by webhost
Thanks for your quick reply, this is how I am trying to run it

http://forums.realwebhost.net/newnews.pl

Is this wrong?

Sort of...

Although several people successfully ran it this way, the script was not written with this intention.

Before you try this, you should run it from the command line / telnet which will allow you to easily see any errors.

If you don't have telnet access, then you need to make sure you have full logging turned on for your webserver and study those logs after running it the way you have been trying. The most common errors you will see will be missing modules. Do a search in this thread for posts by NgtCrwlr. He posted a little script that will list the modules you have installed.

[webhost]

ok thanks, Here is the path to my modules. I believe all are loaded.

http://realwebhost.net/cgi-sys/perldiver.cgi

Joey

[fastforward]

Originally posted by webhost
http://realwebhost.net/cgi-sys/perldiver.cgi

It looks like you have two versions of Perl conflicting. The executables are 5.005 but the modules are 5.6 :confused: Is this what you expected to see?

[webhost]

This is what my host has loaded. Can this still work?

I hope so.

Joey

[webhost]

this is how I have newnews.pl setup, I left password out.
Do I need to change any of this?


#!/usr/bin/perl
# Usenet Gateway hack for vBulletin 2.0
#
# $Id: newnews.pl,v 20010528 2001-05-28 00:29:14-05 paul Exp $
#
# This script creates a gateway to selected usenet newsgroups and maintains
# an historical mirror of articles in your vBulletin forums.
#
# This script is free. You are welcome to do whatever you want with it as
# long as you don't try and sell it or claim it as your own.
# The copyright is retained by the author.
#
# Author:
# fastforward (Paul)
# contact me via the vBulletin forums at http://vbulletin.com
#
#

# uncomment the following line if you installed perl modules to a
# non-standard directory
# use lib "/usr/home/local/lib/perl5/site_perl/5.005";

my $db_name="realweb_forums"; # realweb_forums
my $db_host="localhost"; # localhost
my $db_username="realweb_admin"; # realweb_admin
my $db_password=""; #
my $pid_file="newnews.pid";

[webhost]

can someone help please I am getting this error when I run the script.

[/home/realweb/www]# ./newnews.pl
Global symbol "$db" requires explicit package name at ./newnews.pl line 611.
Global symbol "$db" requires explicit package name at ./newnews.pl line 612.
Execution of ./newnews.pl aborted due to compilation errors

Joey

[webhost]

ok I got it to go through Now I get this when it is done, but nothing shows up on my board.

[/home/realweb/www]# ./newnews.pl
Purging usenet articles...
Purging usenet threads older than 30 days from MSNEWS... OK
Purge completed.

[fastforward]

Originally posted by webhost
ok I got it to go through Now I get this when it is done, but nothing shows up on my board.

[/home/realweb/www]# ./newnews.pl
Purging usenet articles...
Purging usenet threads older than 30 days from MSNEWS... OK
Purge completed.
Assuming you have set up the forum to usenet group mappings correctly, the most common reason for posts not showing is that the posts you downloaded are older than 30 days. This means they are being purged as soon as you import them. Try turning the 'AutoPurge' option off in the control panel.

[webhost]

ok did it we will see what happens. I tried to setup a cron job today earlier to run script every 20 minutes but got a error back back stating it could not run file.
Do you mind if I ask what is the string you use for cron, I can try that and see if that works.

Joey

[webhost]

forgot to mention this is the way I have one newsgroup set up.

newsgroup name alt.php
server name msnews.microsoft.com

is this correct?

Joey

[fastforward]

Originally posted by webhost
Do you mind if I ask what is the string you use for cron, I can try that and see if that works.

This is what I use to run the script every 2 hours

0 0,2,4,6,8,10,12,14,16,18,20,22 * * * nice -10 /home/dbforums/newnews.pl >/dev/null 2>&1

You'll have to make sure that the user the cron job runs under hs permissions to run newnews.pl.

[webhost]

This is what I get

[/home/realweb/public_html]# ./newnews.pl
Connecting to msnews.microsoft.com... Connected
Processing outgoing posts
Getting article batch from alt.php
No new messages in alt.php
inserting new threads from alt.php
checking for orphans...
Processing outgoing posts
Connecting to Msnews.microsoft.com... Connected
Processing outgoing posts
Getting article batch from microsoft.public.frontpage.client
Fetching headers of articles 205 to 304...
Processing headers for article 206... Not available.
Processing headers for article 207... Not available.
Processing headers for article 208... Not available.
Processing headers for article 209... Not available.
Processing headers for article 210... Not available.
Processing headers for article 211... Not available.
Processing headers for article 212... Not available.
Processing headers for article 213... Not available.
Processing headers for article 214... Not available.
Processing headers for article 215... Not available.
Processing headers for article 216... Not available.
Processing headers for article 217... Not available.
Processing headers for article 218... Not available.
Processing headers for article 219... Not available.
Processing headers for article 220... Not available.
Processing headers for article 221... Not available.
Processing headers for article 222... Not available.
Processing headers for article 223... Not available.
Processing headers for article 224... Not available.
Processing headers for article 225... Not available.
Processing headers for article 226... Not available.
Processing headers for article 227... Not available.
Processing headers for article 228... Not available.
Processing headers for article 229... Not available.
Processing headers for article 230... Not available.
Processing headers for article 231... Not available.
Processing headers for article 232... Not available.
Processing headers for article 233... Not available.
Processing headers for article 234... Not available.
Processing headers for article 235... Not available.
Processing headers for article 236... Not available.
Processing headers for article 237... Not available.
Processing headers for article 238... Not available.
Processing headers for article 239... Not available.
Processing headers for article 240... Not available.
Processing headers for article 241... Not available.
Processing headers for article 242... Not available.
Processing headers for article 243... Not available.
Processing headers for article 244... Not available.
Processing headers for article 245... Not available.
Processing headers for article 246... Not available.
Processing headers for article 247... Not available.
Processing headers for article 248... Not available.
Processing headers for article 249... Not available.
Processing headers for article 250... Not available.
Processing headers for article 251... Not available.
Processing headers for article 252... Not available.
Processing headers for article 253... Not available.
Processing headers for article 254... Not available.
Processing headers for article 255... Not available.
Processing headers for article 256... Not available.
Processing headers for article 257... Not available.
Processing headers for article 258... Not available.
Processing headers for article 259... Not available.
Processing headers for article 260... Not available.
Processing headers for article 261... Not available.
Processing headers for article 262... Not available.
Processing headers for article 263... Not available.
Processing headers for article 264... Not available.
Processing headers for article 265... Not available.
Processing headers for article 266... Not available.
Processing headers for article 267... Not available.
Processing headers for article 268... Not available.
Processing headers for article 269... Not available.
Processing headers for article 270... Not available.
Processing headers for article 271... Not available.
Processing headers for article 272... Not available.
Processing headers for article 273... Not available.
Processing headers for article 274... Not available.
Processing headers for article 275... Not available.
Processing headers for article 276... Not available.
Processing headers for article 277... Not available.
Processing headers for article 278... Not available.
Processing headers for article 279... Not available.
Processing headers for article 280... Not available.
Processing headers for article 281... Not available.
Processing headers for article 282... Not available.
Processing headers for article 283... Not available.
Processing headers for article 284... Not available.
Processing headers for article 285... Not available.
Processing headers for article 286... Not available.
Processing headers for article 287... Not available.
Processing headers for article 288... Not available.
Processing headers for article 289... Not available.
Processing headers for article 290... Not available.
Processing headers for article 291... Not available.
Processing headers for article 292... Not available.
Processing headers for article 293... Not available.
Processing headers for article 294... Not available.
Processing headers for article 295... Not available.
Processing headers for article 296... Not available.
Processing headers for article 297... Not available.
Processing headers for article 298... Not available.
Processing headers for article 299... Not available.
Processing headers for article 300... Not available.
Processing headers for article 301... Not available.
Processing headers for article 302... Not available.
Processing headers for article 303... Not available.
Processing headers for article 304... Not available.
Processing article batch...
Requested 0 messages... 0 not available or rejected.

inserting new threads from microsoft.public.frontpage.client
checking for orphans...
Processing outgoing posts

Joey

[webhost]

Ok fastforward here goes. I have run from shell the script about 15 times . I still do not have any posts downloaded still states they are not available. I can see them in outlook express though so I know there is messages there. I did however post a thread to one of the forums and it did upload it and it did show up on the newsgroup in outlook express. So that is working. Waht can I do about receiving messages?

Joey

[fastforward]

That's because your news server is not reporting the message number correctly. The first time the script is ran, it requests the first message number from the news server. Your news server is obviously reporting zero. This causes the script to start at number zero. As you can see, those messages are not available.

Access the news server using a news reader client such as xnews (http://xnews.3dnews.net/) and make a note of the earliest message number in the groups you are interested in. Then manually set the message numbers via the control panel.

[NgtCrwlr]

Ah hah, activity in the usenet gateway channel, this is a good sign. :)

Paul, dare I ask how the next version is coming?

Please don't forget to work on the moderated post glitch. I'm still not letting my members use the gateway until I can moderate the usenet postings.

Also, please double check the HTML::Entities routine again, I'm still having problems with unescaped characters in the subjects. Everything is working fine on vBulletin's end it's only on the incoming posts from newnews.pl that there is a problem.

If you are too busy to get to this for a while yet, please post a few hints on where in vB or newnews that the moderated post fix would go. I will gladly hack away at it, I just need to be led by the nose a bit. :D

Cheers!

[fastforward]

Originally posted by NgtCrwlr
Ah hah, activity in the usenet gateway channel, this is a good sign. :)

Paul, dare I ask how the next version is coming?

Please don't forget to work on the moderated post glitch. I'm still not letting my members use the gateway until I can moderate the usenet postings.

Also, please double check the HTML::Entities routine again, I'm still having problems with unescaped characters in the subjects. Everything is working fine on vBulletin's end it's only on the incoming posts from newnews.pl that there is a problem.

If you are too busy to get to this for a while yet, please post a few hints on where in vB or newnews that the moderated post fix would go. I will gladly hack away at it, I just need to be led by the nose a bit. :D

Cheers!
I've added a basic x-noarchive handler. It simply gives you the option to ignore them and not download them. The problem is, this will cause problems with following posts not being included due to the issues mentioned a few days ago. I'm trying to figure the easiest way to handle this with minimum changes.

All outgoing posts will now honour the vB moderation routines, ie. no post will be posted if it is pending moderation.

As for the escaped characters, have you checked these posts using a normal newsreader? You may find they look the same. if that is the case, there's not much the script can do.

In conclusion, when I fix the problem of threads being 'stopped' when a post is rejected, I'll release the next version. I have every intention on working on it this week.

[webhost]

Well finally got to say it fetched 6 articles but they don't show up anywhere on my board. Don't know where they went. Fetched article 200 through 205 and then 206 through 298 were not available. I checked on forum under the forum I had setup nothing, looked in phpmyadmin nothing. I am able to send threads to newsgroups from board fine I just can not receive any from them. Maybe someone could help me with this tomorrow. Been working on it all day and my eyes are tired. Any ideas drop me a line.
Good night

I'm wopped/

Joey

[NgtCrwlr]

Originally posted by fastforward
As for the escaped characters, have you checked these posts using a normal newsreader? You may find they look the same. if that is the case, there's not much the script can do.

In conclusion, when I fix the problem of threads being 'stopped' when a post is rejected, I'll release the next version. I have every intention on working on it this week.

Excellent news Paul, I have a smile ear to ear. :)

On the entities deal, all we need is to parse the subject line so that it is stored with escaped ampersands, quotes, greater/less than symbols, etc. I looked at newnews.pl and only see one entry for entities.

around line 309
decode_entities($f); decode_entities($s);


Hehehe, while I was typing this out, I thought... get off your lazy duff and figure it out. I think I have done just that, I'm off to test my therory. Be back in a flash with the results.

[NgtCrwlr]

Well, my bright idea turned dim. I know I'm right track, but my Perl skills are just not up to this task. At least not without a hundred or so hours of head scratching and testing.

I thought it would be as easy as changing the decode to encode in this line:
decode_entities($f); decode_entities($s);

but for some reason it loops or repeats a bunch of times. You can see the output here:
http://www.mis-forums.com/vbbs/showthread.php?goto=lastpost&threadid=17939

Then I started to look at newnews closer and realized that not only do incoming subject titles and poster names need to be encoded but they will need to be decoded before sending them back to USENET (I think).

Unfortunately, as we talked about earlier this is a problem of the way vB stores user names and thread/post titles/subjects. If vB stored the raw data (&,",<,>) and filtered it before displaying it like they do in all other data fields there would not be a problem.

This would go unnoticed for a long time because IE5 seems to digest the bad characters without problems, however, it is way bad HTML style to have &'s and greater/less than symbols where they don't belong, and the pages will never come close to validating.

Damn, I'm the bearer of bad news again! :(
I wish I could help more.

[webhost]

Paul I tried to copy and paste from shell but having problem with that. I have attached a zip file on what error I get. I download articles and it states they are downloading and then I get the attached error when it completes. The messages show up in my phpmyadmin but not on my board. It does however add a topic on my board with this title: Using an online form to enter data in to a local database. and also this one Tables set to 100% width
but only a title nothing in threads when you click on them. I get the samw thing every time I run script. my site is at http://forums.realwebhost.net look at frontpage on bottom.

The zip file should show you the error hopefully.

Hope you have time to look at it. This is what my host thinks the probelm is. I don't know need help please.

it's causing an SQL error. I think it's probably due to some
> characters not being escaped correctly. Tell the developer to try
> and auto escape all the characters contained in the pagetext prior
> to it attempting an insert.

Hope you understand that. Joey

[fastforward]

All characters are already escaped before inserting into the database. If they weren't, nothing would work for anybody :)

If you look at your error message, you will see the threadid is null. This is what's causing it to fail.

You need to make sure you have the latest DBI/DBD modules from CPAN. Older versions do not support the msql_insertid function correctly.

[webhost]

Does this correct for a cron job for the script to run it very 20 minutes?

*/20 * * * * /home/realweb/newnews.pl

joey

[NgtCrwlr]

This one fires every 20 minutes
0,20,40 * * * * /your/pathto/newnews.pl >/your/pathto/cron_usenet.log

This one does its work twice an hour
0,30 * * * * /your/pathto/newnews.pl >/your/pathto/cron_usenet.log

Do you see the idea here?
If you want it to run at 12 mins past the hour and then at 43, 47, and 52 minutes past the hour it would be
12,43,47,52 * * * * /your/pathto...

This part of the line
>/your/pathto/cron_usenet.log
saves the output to the the file cron_usenet.log
You can then download cron_usenet.log and see if everything worked properly. Name the file anything you want, just make sure the path is correct for your server.

I hope this helps.

[webhost]

Thanks for ther help guys. PAUL I have live data now in the forum, updated the module as you requested. It works?

Great HACK.

Thanks Crawler for the cron data.

Paul what server do you pull the php and mysql on your site from?

Joey

[NgtCrwlr]

You're welcome Joey, I'm happy you got it working. Now fasten your seatbelt for the disk space ride. You're gonna need a *lot*.

The fun has just begun! :D

[webhost]

thanks

fastened

[fastforward]

Originally posted by webhost

Paul what server do you pull the php and mysql on your site from?

Not sure I understand the question. Do you mean where do I download MySQL & PHP from or are you asking what or where is the server that houses my site?

[webhost]

where you download php, perl, mysql newsgroups from?

Joey

[fastforward]

Originally posted by webhost
where you download php, perl, mysql newsgroups from?

www.usenetserver.com

[webhost]

Script was running fine but it ran again from cron and it ran complete but it showed errors stating something about duplicate entries. Any idea?

Joey

[webhost]

Right now I am just looking for a free plan. Only need php,perl, and mysql for now. Later On I wouldn't mind paying for a service. Just getting started.

Joey

[webhost]

Why does this show up in all my posts?

{usenetindent}>-----

Joey

[fastforward]

Originally posted by webhost
Why does this show up in all my posts?

{usenetindent}>-----

Joey
My guess is you have the 'indent quotes' option turned on but didn't add the required replacement variable.

Have a look at item 6 in the instructions.

[webhost]

call me a dumb a+++ but I can not find it/

Joey

[fastforward]

6. By default, the usenet quotes are color coded blue, green & dark red, according to the level of the quote (up to three levels). You can override these colours by adding the following replacement variables to your current style set as follows:

{usenetquotecolor} REPLACED WITH blue
{usenetquotecolor2} REPLACED WITH green
{usenetquotecolor3} REPLACED WITH darkred

The above are not necessary if you are happy with the default colours of red, green & blue.

Additionally, if you would like quotes to be indented according to the quote level, you can add the following replacement variable to your current styleset.

{usenetindent} REPLACED WITH &nbsp;&nbsp;

The indent does NOT happen by default so you will have to add this variable if you want it.

Create a replacement variable for {usenetindent} as shown above, using the normal vB control panel.

[NgtCrwlr]

Go to vB control panel...
select Usenet/Options...
scroll to the third table...
there you will find -- Italics, Indents, Colours, etc.

Make sure - Indents is set to (No)

[webhost]

ok I think I got it I had the variable created by the time I got crawler's post. If you would look at this link scroll new the first reply and tell me if the text should look like this and also this is showing up.

>


http://forums.realwebhost.net/showthread.php?threadid=109

Thanks Joey

[NgtCrwlr]

The page looks fine from my end.

[webhost]

got any idea what would be doing this?

Script was running fine but it ran again from cron and it ran complete but it showed errors stating something about duplicate entries. Any idea?

Joey

Thanks Again Guys

Wouldn't have been able to do it without you guys.

[webhost]

crawler something like this for 20 minute cron job?

0,20,40 * * * * /home/realwebhost.net/newnews.pl > /home/realwebhost.net/cron_usenet.log

[fastforward]

Originally posted by webhost
got any idea what would be doing this?

Script was running fine but it ran again from cron and it ran complete but it showed errors stating something about duplicate entries. Any idea?

Joey

Thanks Again Guys

Wouldn't have been able to do it without you guys.
Is it the word table? There were a few posts recently about this. Have a look back through the last 6 pages or so. This table seems to get corrupted very easily with or without the usenet hack. It seems to happen on shared servers which may indicate a problem with the number and frequency of inserts and selects. However, the problem doesn't seem to happen much with the searchindex table which probably receives even more queries.

Your best bet is to empty it out and re-index. If it continues to happen then you will have no choice but to disable indexing for the usenet posts.

[NgtCrwlr]

Originally posted by webhost
got any idea what would be doing this?
Common Joey, how the heck are we supposed to know what *this* is? You need to include what the error is if you expect someone to help.
Script was running fine but it ran again from cron and it ran complete but it showed errors stating something about duplicate entries. Any idea?
Same as above... until we know exactly what the error is how can you expect an accurate answer.
Thanks Again Guys

Wouldn't have been able to do it without you guys.
This may sound a bit sarcastic but sometimes the truth bites... You most likely would have got it running by yourself if you were being charged $150/hr. Every question you have asked you could have figured out by your self... really :)

You ask if this will work... try it and if it doesn't work figure out why not. With the examples I posted earlier you should easily get this working.
0,20,40 * * * * /home/realwebhost.net/newnews.pl > /home/realwebhost.net/cron_usenet.log

If you pretend to yourself that you will have to pay for an answer, you will write a much better question (details). Not as good as if you really had to pay but better nonetheless.

I'm more than willing to help anyone that tries to help themselves. Although, I do have my limits. ;)

No hard feelings, I hope.

[webhost]

Understand No hard feelings. I will run it for a few wweks and then come back if I have any problems area's and then ask, but will try to figure out first.

Thanks Again

Joey

[mood]

Hi All, Hi FF!

New to the Usenet Gateway and loving it BUT:

When picking up msgs from my news server, the Perl script shows it retrieving fine though the posts only show up in VB if I change the counter -1 msg or whatever.

At first I though it could be my news server being misconfigured but tried two others and the problem persisted.

Any chance of fixing this?

Any helpful hint is VERY welcome.

Thanks to all

Alex (aka Mood)

[fastforward]

Originally posted by Mood
Hi All, Hi FF!

New to the Usenet Gateway and loving it BUT:

When picking up msgs from my news server, the Perl script shows it retrieving fine though the posts only show up in VB if I change the counter -1 msg or whatever.

At first I though it could be my news server being misconfigured but tried two others and the problem persisted.

Any chance of fixing this?

Any helpful hint is VERY welcome.

Thanks to all

Alex (aka Mood)
I have no idea what you mean. What counter are you talking about? If you mean the 'last message number' then that has absolutely nothing to do with posts suddenly showing up in your forums. The message number is used for retrieving posts from the server and as you stated, that is working OK.

It is the message ID that determines when and if an article is posted to the forums. This ID is unique across all of Usenet and should never be tampered with.

[mood]

[QUOTE]Originally posted by fastforward

I have no idea what you mean. What counter are you talking about? If you mean the 'last message number' then that has absolutely nothing to do with posts suddenly showing up in your forums. The message number is used for retrieving posts from the server and as you stated, that is working OK.

It is the message ID that determines when and if an article is posted to the forums. This ID is unique across all of Usenet and should never be tampered with.

[NgtCrwlr]

I have a suspicion that you are not selecting the right message ID for your news server. I'm hoping that you use Outlook, if so here is a capture showing the number you are looking for.

Pick a message around the date you want to start pulling.

Right-click on the message title
Select properties
Select details

http://www.makeitsimple.com/images/misc/outlook_usenet.gif

[chrome]

A bit of background first:

Solaris 8 on an E420R
Apache 1.3.20
PHP 4.0.5
MySQL 3.23.39

I've got vB installed fine and it works great. I've downloaded your hack and it works (you need a config option for the path to ps - on solaris the ps you want lives in /usr/ucb - thats where all the BSD compatible binaries live) to the extent that it will allow me to use the control panel, and the newnews.pl script runs without any warnings or obvious errors, happily synching away to the INN server.

However, the bodies of the messages are blank. Any ideas?

vB and the Usenet Hack are installed stock - I've done no customisation to either other than the bare minimum to get them to work properly. I've not got any anti-spam filters or anything in place. Its just borked :)

[fastforward]

Hmmm.. this sounds like a tricky one :(

If you are getting the headers correctly, then we know it's talking to the INN server correctly and understands how to retrieve header information by message number.

The next thing the script does is to run those headers through the spam filter and if it decides it's an article we want, it requests the body from the news server using the message number and the 'body()' method from perl NNTP module. Assuming you have no replacements set, this must be the bit that is failing to retrieve the article from the server.

Why it's failing, I have no idea. Have you got another news server you can try. That way you can determine if it's an INN specific thing. You might want to make sure you have the latest NNTP module from CPAN aswell.

[chrome]

Originally posted by fastforward
Hmmm.. this sounds like a tricky one :(

...

Why it's failing, I have no idea. Have you got another news server you can try. That way you can determine if it's an INN specific thing. You might want to make sure you have the latest NNTP module from CPAN aswell.

Yeah, I installed perl 5.6.1 and all the modules fresh, as a precaution. Unfortunately that machine can't get out to the net easily, so it doesn't really have access to another news server I can test against.

Thats not a major problem though. I was simply wondering if anyone had bumped into this one before.. I'm going to hack some debugging into your code and see what it *thinks* it's inserting into the DB =). A look at the database would probably be helpful, too.

BTW, have you thought about using a newsfeed instead of a poll method to populate the database? I wrote a perl script (still have it if you want it) that plugged into an INN server and pushed postings into a mysql database. It's spawned by INN when it starts, and INN pushes it an id of every message that INN gets sent (whether by an external feed, or a client) and then the script uses an INN supplied command to turn that id into a path to the physical location of the posting. The script reads it into an NNTP::Article structure and gets all the headers/body and dumps it into relevant fields in a MySQL database.

It's a bit broken atm, no checking for SQL statements in the body/headers etc :eek: but it works.

I was originally going to turn it into a vB clone, but then I found your hack. :D Much nicer to get vB to do all the hard work thats already been done, eh?

[fastforward]

Originally posted by chrome
BTW, have you thought about using a newsfeed instead of a poll method to populate the database? I wrote a perl script (still have it if you want it) that plugged into an INN server and pushed postings into a mysql database. It's spawned by INN when it starts, and INN pushes it an id of every message that INN gets sent (whether by an external feed, or a client) and then the script uses an INN supplied command to turn that id into a path to the physical location of the posting. The script reads it into an NNTP::Article structure and gets all the headers/body and dumps it into relevant fields in a MySQL database.
The reason I did it this way was that most people don't have either a full newsfeed or their own news server. Most of us are using a personal news account such as Supernews. There are issues of missing posts and problems with different character sets that could probably be handled more easily if the news server was dumping things raw into a database for us to process later so I would be very interested in your script if you'd like to share it :)

[chrome]

Originally posted by fastforward

The reason I did it this way was that most people don't have either a full newsfeed or their own news server. Most of us are using a personal news account such as Supernews. There are issues of missing posts and problems with different character sets that could probably be handled more easily if the news server was dumping things raw into a database for us to process later so I would be very interested in your script if you'd like to share it :)

http://www.stupendous.net/news2mysql-0.1.tar.gz

I hope you find it useful.

I notice that you don't use News::Article for parsing of the articles? You'd probably find you would have better luck using that to handle all the character set issues and to avoid using home-grown regex's. I'm sure it uses regex's internally (not bothered to look) but why reinvent the wheel, eh?

How do Supernews feel about you polling their servers every 30 minutes? :)

[chrome]

Hrm...

The postings are definately in the database, and your newnews.pl is definately pulling out the body fine, its just that vB is not displaying it at all. Very strange :confused:

Feh.

Do you know if there is any debugging I can turn on in vB that will show extended information about the postings when viewing threads? its almost as if the posting isn't being associated with the thread properly.

In fact I described my problem incorrectly. The threads are being created but individual posts are not showing up in the threads - it's not the body of the message at all - Doh...

[chrome]

For some reason newnews.pl isn't assigning a threadid to any of the posts that it inserts into the DB. More investigation needed I think =)

[fastforward]

Originally posted by chrome
For some reason newnews.pl isn't assigning a threadid to any of the posts that it inserts into the DB. More investigation needed I think =)
Your DBI or DBD module is out of date. Read back a few pages and you will see other references to this. Older versions of these modules do handle the mysql_insertid() function correctly.

[webhost]

Here's is a news server that I found.

ccnews.thu.edu.tw

Joey

[chrome]

Originally posted by fastforward

Your DBI or DBD module is out of date. Read back a few pages and you will see other references to this. Older versions of these modules do handle the mysql_insertid() function correctly.

DBI-1.18
Msql-Mysql-modules-1.2216
perl 5.6.1

those are the latest afaics

What versions are you using to develop with?

[fastforward]

Originally posted by chrome


DBI-1.18
Msql-Mysql-modules-1.2216
perl 5.6.1

those are the latest afaics

What versions are you using to develop with?
I'm using those versions.

The problem you are seeing is due to something not handling the mysql_insertid() function.

Others had the same problem and solved it by upgrading their modules. Perhaps it's a MySQL issue. I'm not sure as I have never encountered the problem.

[chrome]

Oh well. :(

I did write a small test perl script that basically inserted a row into a table and tried to use the mysql_insertid function to get the id, and it doesn't work, so I'm confident it's entirely my problem, and not your code at all ;).

I reinstalled my Msql-Mysql-modules and when I did a make test, I noticed some failures. Hrm. Something about my Solaris 8 sparc platform I think. I'm going to try upgrading the compiler to gcc 2.95.3 and seeing if that helps (rebuilding all the DBI related modules).

If not, I'll start bugging the DBI/MySQL developers :D

Thanks for the help anyway. Good job on the hack, in any case :)

chrome.

[webhost]

Ok I have about 15 newsgroups online. I reindexed and the turned ON usenet index. Got the same problem I had the other day with duplicate entries. I have turned back off,

All newsgroups are working fine except a small probel with 2 of the them. I run newnews.pl and watch it download messages for these 2 groups, example tonight each one downloaded 24 messages each. I go to my forums and they still show zero in the count. I went and looked at the orphan files for each but they also are empty. The 2 groups or comp.mail.smail and comp.mail.eudora.
Does anyone where these messages might be going?

Joey

[webhost]

Sorry guys I think I found it. It might have worked if someone had posted to either one of those groups in the last month. They must be dead or something

Joey

[webhost]

Did anyone ever come with a answer to somehow minimize the amount of orphans not be posted. I have been getting a alot of them. Some of them I see 4 or 5 the same ones all re: some of them are replies to a known thread that is already showing up on my board. Any ideas or fix's

Joey

[boatdesign]

Wow - this is exactly what I've been looking for! Thank you, thank you, thank you!

I am very excited about the possiblity of integrating 2 usenet newgroups with my vBulletin forums, but I'm worried about doing anything which might endanger my hosting account.

1.) Can this script be run on a virtual/shared host? (300 other hosting accounts per RH Linux server, P3 800, 1 GB Ram)
Is anyone running this on a shared host, or is it for dedicated servers only?

2.) What kind of server resources would be required if I was looking at including 2 usenet groups which currently have 100 messages per day max?

3.) What happens if connection to the news server hangs? (lingering processes?)

Also, I am a little worried about the possibility of being targeted by a spammer who could use my forums to spam usenet and thus get my hosting account suspended.

4.) Is there any type of post rate limiter designed in?

5.) Is there an option to allow only moderated posts to usenet (or maybe an option to have them appear in the forum before being approved for posting to usenet?)

Thanks very much - this is a fantastic option. I just hope that I will be able to use it.

[hhdream]

I just installed the script - wow!
I am using Cyrilic encoding but on my forum i have WIN-1251-codepage and usenet-topics in russian are in KOI-8. How I can convert it to 1251?

[Joe]

Can i get this in .zip format? Im having problems with the tar.gz format...

[boatdesign]

<a href="http://www.winzip.com/" target="_blank">http://www.winzip.com/</a> unpacks it without any trouble...

[fastforward]

Originally posted by hhdream
I just installed the script - wow!
I am using Cyrilic encoding but on my forum i have WIN-1251-codepage and usenet-topics in russian are in KOI-8. How I can convert it to 1251?
I saw Ed post a reply about a similar problem just a three or four days ago. It involved editing the vB code somewhere. Your edits will need to be a little more complicated though as you only want to make changes to forums that contain usenet threads.

The character set options in the usenet control panel are just for the quoted printable character conversions.

Sorry I can't offer any more help.

[hhdream]

I have found how to convert KOI8 to CP1251

Modules to install - Convert::Cyrillic; Unicode::Map8

Code to add:
use Convert::Cyrillic;

Code to change:
$wd->decode(YOURTEXT) replace to
Convert::Cyrillic::cstocs ('koi8','win', YOURTEXT)

for example
$$art{body} = $wd->decode($$art{body})
$$art{body} = Convert::Cyrillic::cstocs ('koi8','win', $$art{body});


:D

[george_proost]

Just downloaded and activated basic setup. Exactly what I need.

HOWEVER... I get a

parse error on usenet.php on line 378

also

Warning: GLOBAL variable declaration meaningless in main() scope in /xx/xx/xx/admin/usenet.php on line 353


trying to do the administration ??

are these known errors. Or probably, have I missed something.


Could it have something to do with needing to insert xtra \\s in some SQL inserts?


Does the newnews.pl need to run first? before I do admin.


:confused:

[webhost]

I am installing some of the options with this hack. I have installed 3 or 4 of them. 2 questions; First
I installed the stat's hack in stats.php, but when I go to the admin control panel and view stat's it does not show usenet stat's I then looked in drop down menu and there was not a selection to choose them. I also installed the hack that separtes the board post from the usenet post. Would this have something to do with them not showing up on the stat's page or to do I have to place a command in one of templates for this?
Second Question.
I want to add the last active thread option, Before I do I would like someone to look at code below and make sure I'm putting it in the right place. Area's I am going to place are in bold and state before this to the side.

8) Populating of last active thread in the forum table. This can then be referenced in templates by using $forum[lastactivethread].
a) admin/functions.php - line 1022

if ($newwords) {
$newwords=trim($newwords);
$insertwords="(NULL,'".str_replace(" ","'),(NULL,'",addslashes($newwords))."')";
$DB_site->query("INSERT IGNORE INTO word (wordid,title) VALUES $insertwords");
$selectwords="title IN('".str_replace(" ","','",addslashes($newwords))."')";
$DB_site->query("INSERT IGNORE INTO searchindex (wordid,postid) SELECT DISTINCT wordid,$postid FROM word WHERE $selectwords");
} ===before this====

if ($newtitlewords) {
$newtitlewords=trim($newtitlewords);
$insertwords="(NULL,'".str_replace(" ","'),(NULL,'",addslashes($newtitlewords))."')";
$DB_site->query("INSERT IGNORE INTO word (wordid,title) VALUES $insertwords");
$selectwords="title IN('".str_replace(" ","','",addslashes($newtitlewords))."')";
$DB_site->query("REPLACE INTO searchindex (wordid,postid,intitle) SELECT DISTINCT wordid,$postid,1 FROM word WHERE $selectwords");
}
}===before this===

// ###################### Start makeforumjump #######################

b) admin/functions.php - line 1577

$DB_site->query("DELETE FROM thread WHERE threadid='$threadid'");
$DB_site->query("DELETE FROM thread WHERE open=10 AND pollid='$threadid'"); // delete redirects
$DB_site->query("DELETE FROM threadrate WHERE threadid='$threadid'");
$DB_site->query("DELETE FROM subscribethread WHERE threadid='$threadid'");
}
}===before this====
// ###################### Start delete post #######################

c) admin/functions.php - line 1599

$DB_site->query("DELETE FROM searchindex WHERE postid=$postid");
$DB_site->query("DELETE FROM post WHERE postid='$postid'");
}
}===before this===

// ###################### Start make login code #######################

d) admin/functions.php - New function at end of file.

//searches expire after a week:
$DB_site->query("DELETE FROM search WHERE dateline<".(time()-(7*24*60*60)));
}

// bye bye!
}===before this====
?>

This is where I put the hack in stats.php

$from = mktime(12,0,0,$frommonth,$fromday,$fromyear);

//START USENET HACK
if (($type == "post" || $type == "thread") && $showusenetpostsinstats==0) {
$stats = $DB_site->query("SELECT COUNT(*), DATE_FORMAT(FROM_UNIXTIME($field),'$sqlformat') AS timeframe, MAX($field) FROM $table WHERE $field > '$from' AND $field < '$to' AND isusenetpost=0 GROUP BY timeframe ORDER BY $field $sort");
} else { $stats = $DB_site->query("SELECT COUNT(*), DATE_FORMAT(FROM_UNIXTIME($field),'$sqlformat') AS timeframe, MAX($field) FROM $table WHERE $field > '$from' AND $field < '$to' GROUP BY timeframe ORDER BY $field $sort"); }
//END USENET HACK
unset($totals);
unset($dates);
unset($sum);
while ($stat = $DB_site->fetch_array($stats)) {
$totals[] = $stat[0];
if ($stat[0]>$max) {
$max=$stat[0];

[v0n]

Houston, we have a problem...

Just moved my forums to new server, installed perl 5.6, MySQL 3.23.38, all modules required, installed usenet hack and...

- VBB shows thread titles but no posts inside them.
- All usenet posts retrieved from the server in database have threadid=0

Here's what I did so far:

Manualy compiled all the modules once again with versions as posted in this thread
Once again installed Bundle::CPAN and all the required modules matching CPAN version.

Still the same - all posts are inserted to database with threadid=0 and don't appear under threads in VBB...

[fastforward]

Originally posted by webhost
I am installing some of the options with this hack. I have installed 3 or 4 of them. 2 questions; First
I installed the stat's hack in stats.php, but when I go to the admin control panel and view stat's it does not show usenet stat's I then looked in drop down menu and there was not a selection to choose them. I also installed the hack that separtes the board post from the usenet post. Would this have something to do with them not showing up on the stat's page or to do I have to place a command in one of templates for this?
Second Question.
I want to add the last active thread option, Before I do I would like someone to look at code below and make sure I'm putting it in the right place. Area's I am going to place are in bold and state before this to the side.

8) Populating of last active thread in the forum table. This can then be referenced in templates by using $forum[lastactivethread].
a) admin/functions.php - line 1022

if ($newwords) {
$newwords=trim($newwords);
$insertwords="(NULL,'".str_replace(" ","'),(NULL,'",addslashes($newwords))."')";
$DB_site->query("INSERT IGNORE INTO word (wordid,title) VALUES $insertwords");
$selectwords="title IN('".str_replace(" ","','",addslashes($newwords))."')";
$DB_site->query("INSERT IGNORE INTO searchindex (wordid,postid) SELECT DISTINCT wordid,$postid FROM word WHERE $selectwords");
} ===before this====

if ($newtitlewords) {
$newtitlewords=trim($newtitlewords);
$insertwords="(NULL,'".str_replace(" ","'),(NULL,'",addslashes($newtitlewords))."')";
$DB_site->query("INSERT IGNORE INTO word (wordid,title) VALUES $insertwords");
$selectwords="title IN('".str_replace(" ","','",addslashes($newtitlewords))."')";
$DB_site->query("REPLACE INTO searchindex (wordid,postid,intitle) SELECT DISTINCT wordid,$postid,1 FROM word WHERE $selectwords");
}
}===before this===

Correct.

// ###################### Start makeforumjump #######################

b) admin/functions.php - line 1577

$DB_site->query("DELETE FROM thread WHERE threadid='$threadid'");
$DB_site->query("DELETE FROM thread WHERE open=10 AND pollid='$threadid'"); // delete redirects
$DB_site->query("DELETE FROM threadrate WHERE threadid='$threadid'");
$DB_site->query("DELETE FROM subscribethread WHERE threadid='$threadid'");
}
}===before this====

Correct.

// ###################### Start delete post #######################

c) admin/functions.php - line 1599

$DB_site->query("DELETE FROM searchindex WHERE postid=$postid");
$DB_site->query("DELETE FROM post WHERE postid='$postid'");
}
}===before this===

Correct.

// ###################### Start make login code #######################

d) admin/functions.php - New function at end of file.

//searches expire after a week:
$DB_site->query("DELETE FROM search WHERE dateline<".(time()-(7*24*60*60)));
}

// bye bye!
}===before this====
?>

WRONG!
It is a NEW function. Add it after the brace but before the php close

This is where I put the hack in stats.php

$from = mktime(12,0,0,$frommonth,$fromday,$fromyear);

//START USENET HACK
if (($type == "post" || $type == "thread") && $showusenetpostsinstats==0) {
$stats = $DB_site->query("SELECT COUNT(*), DATE_FORMAT(FROM_UNIXTIME($field),'$sqlformat') AS timeframe, MAX($field) FROM $table WHERE $field > '$from' AND $field < '$to' AND isusenetpost=0 GROUP BY timeframe ORDER BY $field $sort");
} else { $stats = $DB_site->query("SELECT COUNT(*), DATE_FORMAT(FROM_UNIXTIME($field),'$sqlformat') AS timeframe, MAX($field) FROM $table WHERE $field > '$from' AND $field < '$to' GROUP BY timeframe ORDER BY $field $sort"); }
//END USENET HACK
unset($totals);
unset($dates);
unset($sum);
while ($stat = $DB_site->fetch_array($stats)) {
$totals[] = $stat[0];
if ($stat[0]>$max) {
$max=$stat[0]; [/B]
The stats option doesn't actually add a new option. I think there may be a typo. I believe it actually shows the usenet posts UNLESS you apply this option. The next version will provide more options for stats.

[fastforward]

Originally posted by v0n
Houston, we have a problem...

Just moved my forums to new server, installed perl 5.6, MySQL 3.23.38, all modules required, installed usenet hack and...

- VBB shows thread titles but no posts inside them.
- All usenet posts retrieved from the server in database have threadid=0

Here's what I did so far:

Manualy compiled all the modules once again with versions as posted in this thread
Once again installed Bundle::CPAN and all the required modules matching CPAN version.

Still the same - all posts are inserted to database with threadid=0 and don't appear under threads in VBB...
Looks like you're having the same problem as chrome. As you know, a couple of people have seen this problem but have fixed it by upgrading their modules. See if you can get in touch with chrome and see if he's found the problem.

[webhost]

fastforward:
ok I got all the code in place for the lastactivethread and I placed on the following location in the forumdisplaybit template but it still not showup on the opening page in my forums. Can you direct me on where I am going wrong below is the script. I have tried $forum, $forum2, and $thread in front of [lastactivethread] but none work.

<tr align="center">
<td bgcolor="#DFDFDF"><img src="images/$thread[newoldhot].gif" border="0" alt=""></td>
<td bgcolor="#DFDFDF">$thread[icon]</td>
<td bgcolor="#F1F1F1" align="left" width="100%"><normalfont>$thread[gotonew] $paperclip$thread[movedprefix]$thread[typeprefix]<a href="showthread.php?s=$session[sessionhash]&threadid=$thread[threadid]">$thread[title]</a></normalfont> <smallfont>$thread[pagenav]</smallfont></td>
<td align="right"bgcolor="#DFDFDF" nowrap><smallfont>

$forum[lastactivethread]

$thread[firstpostdate] $thread[firstposttime]<br>by <b>$thread[postedby]</b></smallfont></td>
<td bgcolor="#F1F1F1"><normalfont>
<a href="javascript:who($thread[threadid])">$thread[replycount]</a>
</normalfont></td>
<td bgcolor="#DFDFDF"><normalfont>$thread[views]</normalfont></td>
$threadrating
<td bgcolor="$backcolor">
<table cellpadding="0" cellspacing="0" border="0"><tr>
<td rowspan="2" nowrap><a href="showthread.php?s=$session[sessionhash]&goto=lastpost&threadid=$thread[threadid]"><img src="images/lastpost.gif" border="0" alt="Go to last post"></a>&nbsp;</td>
<td nowrap><smallfont>$thread[lastreplydate] <smallfont color="#666686">$thread[lastreplytime]</smallfont></smallfont></td>
</tr><tr>
<td nowrap><smallfont>by <a href="member.php?action=getinfo&find=lastposter&threadid=$thread[threadid]"><b>$thread[lastposter]</b></a></smallfont></td>
</tr></table>
</td>
</tr>

[george_proost]

An excellent hack, but as per my previous post. I don't understand where it goes wrong. The admin function does it's job ok... but I get those errors mentioned... any pointers please.
I'm willing of course to do legwork.

I loaded the version from the top, 2 days ago. Perl things in place...etc. Please I need the base to work before I apply the other hacks.

I'm a mainframe guy :o

T.I.A.

George

see the urgency of this need requirement at this site (http://www.mainframeforum.com)

The USENET Forums are currently hidden.

[fastforward]

Originally posted by george_proost
Just downloaded and activated basic setup. Exactly what I need.

HOWEVER... I get a

parse error on usenet.php on line 378

also

Warning: GLOBAL variable declaration meaningless in main() scope in /xx/xx/xx/admin/usenet.php on line 353


trying to do the administration ??

are these known errors. Or probably, have I missed something.


Could it have something to do with needing to insert xtra \\s in some SQL inserts?


Does the newnews.pl need to run first? before I do admin.


:confused:
I have never seen this error before and the usenet.php file has not been changed for months. What does your line 378 say?

[george_proost]

cpfooter();

if ($action=="manual") {
if ($mode=="Delete") {
foreach ($lostmsgid as $i){
$DB_site->query("DELETE FROM usenet_article WHERE msgid='" . addslashes($i) . "'");
}
echo "Message(s) Deleted<br><br>";
}
if ($mode=="Insert into thread") {

//$num_rows = $DB_site->num_rows($threadmsgids);
if ($threadmsgids=$DB_site->query("SELECT msgid,ord FROM post WHERE threadid=$threadid ORDER BY dateline LIMIT 1")){
$thread=$DB_site->fetch_array($threadmsgids);
$threadmsgid = $thread["msgid"];
$ord = $thread["ord"]; // most likely 0
foreach ($lostmsgid as $i){
$DB_site->query("UPDATE usenet_article SET refs='" . addslashes($threadmsgid) . "', ord=$ord, manual=1 WHERE msgid='" . addslashes($i) . "'");
$DB_site->query("INSERT IGNORE INTO usenet_ref (msgid,ref) VALUES ('" . addslashes($i) . "','" . addslashes($threadmsgid) . "')");
}
echo "Messages inserted.<br><br>";
}
else { echo "Thread was not found or not specified. Please go back and select a valid thread.";}
}
if ($mode=="New Threads") {
foreach ($lostmsgid as $i){
$subjects=$DB_site->query("SELECT subject FROM usenet_article WHERE msgid='" . addslashes($i) . "' LIMIT 1");
$subject=$DB_site->fetch_array($subjects);
$subject=$subject["subject"];
$subject = eregi_replace("^Re:[ ]*","",$subject);
$DB_site->query("UPDATE usenet_article SET refs='', ord=0, manual=1, subject='" . addslashes($subject) ."' WHERE msgid='" . addslashes($i) . "'");
}
echo "New threads created<br><br>";
}

[fastforward]

Originally posted by webhost
fastforward:
ok I got all the code in place for the lastactivethread and I placed on the following location in the forumdisplaybit template but it still not showup on the opening page in my forums. Can you direct me on where I am going wrong below is the script. I have tried $forum, $forum2, and $thread in front of [lastactivethread] but none work.
Check in the forum table and see if the lastactivethread column is populated.

You might be better of not implementing this option. Instead, you can use this (http://www.vbulletin.com/forum/showthread.php?s=&threadid=19446) hack. It supports showing the icon aswell. newnews.pl works with it.

[fastforward]

Originally posted by george_proost
cpfooter();

if ($action=="manual") {
if ($mode=="Delete") {
foreach ($lostmsgid as $i){
$DB_site->query("DELETE FROM usenet_article WHERE msgid='" . addslashes($i) . "'");

Don't know what to suggest I'm afraid. Nobody else has reported this problem and your file matches mine. I can only assume it is something specific to your PHP version or configuration.

[george_proost]

foreach clause is not supported in PHP3 :(
now to figure out that standard loopy way
hmmmm.... its only the bottom part...
gotta figure out what it does!!

[fastforward]

Originally posted by george_proost
foreach clause is not supported in PHP3 :(
now to figure out that standard loopy way
hmmmm.... its only the bottom part...
gotta figure out what it does!!
That entire function is for manually inserting orphan replies into threads. It doesn't work at the moment any way. I broke it when I made other changes :) You should be able to safely comment out the lot.

[george_proost]

Comment out the foreach all of them

## foreach ($lostmsgid as $i){

Insert after each of them

for($ix = 0; $ix < sizeof($lostmsgid); $ix++) {
$i = $lostmsgid[$ix];

that should do it .... I'm clean now with the parse error.

[webhost]

lastactivethread column is populated but still nothing. I did however use your recommendation and install other hack with the same results. Here is the forumdisplaybit template, I have at the bottom placed how I setup the 2 tables in your hack. I have also attached my functions.php in a zip format. My site is http://www.realwebhost.net

<tr align="center">
<td bgcolor="#DFDFDF"><img src="images/$thread[newoldhot].gif" border="0" alt=""></td>
<td bgcolor="#DFDFDF">$thread[icon]</td>
<td bgcolor="#F1F1F1" align="left" width="100%"><normalfont>$thread[gotonew] $paperclip$thread[movedprefix]$thread[typeprefix]<a href="showthread.php?s=$session[sessionhash]&threadid=$thread[threadid]">$thread[title]</a></normalfont> <smallfont>$thread[pagenav]</smallfont></td>
<td align="right"bgcolor="#DFDFDF" nowrap><smallfont> $forum[lastactiveiconid] $forum[lastactivethread] $thread[firstpostdate] $thread[firstposttime]<br>by <b>$thread[postedby]</b></smallfont></td>
<td bgcolor="#F1F1F1"><normalfont>
<a href="javascript:who($thread[threadid])">$thread[replycount]</a>
</normalfont></td>

Field Type Length/Set Attributes Null Default
lastactivethread varchar(64) Yes
lastactiveiconid smallint(5) UNSIGNED No 1

[fastforward]

Originally posted by webhost
lastactivethread column is populated but still nothing. I did however use your recommendation and install other hack with the same results. Here is the forumdisplaybit template, I have at the bottom placed how I setup the 2 tables in your hack.
I see one problem in your template, and that is the iconid. iconid only returns the number. You still need to add the image tag and 'icon' part of the name. Here's my template:

------------------------------------------------
<table cellpadding="0" cellspacing="0" border="0" width="100%" id="ltlink">
<tr>
<td rowspan="2" valign="top"><a href="$bburl/showthread.php?s=$session[sessionhash]&goto=lastpost&forumid=$forum[forumid]" title="Go to last post"><img src="images/icons/icon$forum[lastactiveiconid].gif" border="0" alt="">&nbsp;</a></td>
<td align="left"><smallfont><a href="$bburl/showthread.php?s=$session[sessionhash]&goto=lastpost&forumid=$forum[forumid]" title="Go to last post"><font color="#000000">$forum[lastactivethread]</font></a> (<a href="$bburl/member.php?s=$session[sessionhash]&action=getinfo&find=lastposter&forumid=$forum[forumid] title="See $forum[lastposter]s profile">$forum[lastposter]</a>)</smallfont></td>
</tr>
<tr>
<td nowrap><smallfont>$forum[lastpostdate] <font color="#666686">$forum[lastposttime]</font></td>
</tr>
</table>
------------------------------------------------

Try just referencing it just by $lastactiveiconid and $lastactivethread, without the 'forum' bit. If that doesn't work, try $foruminfo[lasticonid] and $foruminfo[lastactivethread]

If that doesn't work, then you're on your own. :) I assume you are using the latest vB and are on php4?

[webhost]

yes vb2.0.1 php4
Thanks I'll try it

Joey

[webhost]

I tried! I think I give up for awhile and try to enjoy the 4th.

Happy July 4th everyone.

Joey

Thanks FastForward

[george_proost]

commented out line with following text

// global $DB_site;


caused a warning in admin
usenet.php ... PHP3 change

admin routine now clean for php3 :)

[fastforward]

Originally posted by george_proost
commented out line with following text

// global $DB_site;


caused a warning in admin
usenet.php ... PHP3 change

admin routine now clean for php3 :)
Are you sure it all still works?

The only thing that can safely be disabled is the [Review/Insert Orphans] option within the 'Groups' section. This is handled by the first block of functions you spoke about. $DB_site is the main database class that handles all database access.

[v0n]

Originally posted by fastforward

Looks like you're having the same problem as chrome. As you know, a couple of people have seen this problem but have fixed it by upgrading their modules. See if you can get in touch with chrome and see if he's found the problem.

Just for the purpose of the excercise I have installed Slackware, MySQL 1.23.39, Perl 5.6 (from slackware pkg distribution) and compiled modules from CPAN (no errors and no forced installs) on yet another machine. And as you may have guessed already - inserted news posts have threadid 0. So it's either down to perl version or the fresh install of vBulletin 2.0.1 doesn't take the mod as good as vB upgraded from betas? :(

[fastforward]

Originally posted by v0n


Just for the purpose of the excercise I have installed Slackware, MySQL 1.23.39, Perl 5.6 (from slackware pkg distribution) and compiled modules from CPAN (no errors and no forced installs) on yet another machine. And as you may have guessed already - inserted news posts have threadid 0. So it's either down to perl version or the fresh install of vBulletin 2.0.1 doesn't take the mod as good as vB upgraded from betas? :(
Mine is installed on a fresh 2.01. I'm using Perl 5.005_03 and always have. I know chrome was using Perl 5.6, how about you?

[shri]

Would it be possible to split this thread up? Its a pain in the arse going through 40 pages of posts to find a keyword or two :)

[george_proost]

Originally posted by fastforward

Are you sure it all still works?

The only thing that can safely be disabled is the [Review/Insert Orphans] option within the 'Groups' section. This is handled by the first block of functions you spoke about. $DB_site is the main database class that handles all database access.

It certainly does still work ( the admin stuff that is ) ... the message was a WARNING only indicating that the GLOBAL had already been defined.

This is correct... it is defined in the calling 'adminfunctions.php' and I think is valid on subsequent deeper calls.

I'm still waiting for my sys-admin to activate the additional perl modules. The rest is ready on my side... hopefully today sometime.

Thanks for your help so far..

[george_proost]

Originally posted by shri
Would it be possible to split this thread up? Its a pain in the arse going through 40 pages of posts to find a keyword or two :)

shri, this list is BIG yes... however it essential to keep it as one.
Take some advice and start from the beginning of the posts. It is VERY important to get a good handle on things and makes the implementation easier.

All persons on this thread have been a great help so far.
I hope to have the USNET Gateway active soon. If I experience problem I know where to come to ! so, split it up,

nooooooooooooooooo thanx

[TheDude]

clear example that vb needs subthread handling ;)

[george_proost]

hmmmmm ... when you start ordering a list like this
hmmmmmmm ... does it not become a forum ..

admin??? a sub-forum structure for recognized and elite projects


>> USENET Gateway (top of the list please)
....

vB can handle this easy ............ :D

I'm sure the mod will split accordinglly --->

eg..
Release download area
The Package Contents
System Requirements
Installation
Problems

[dwh]

Originally posted by fastforward

Is it the word table? There were a few posts recently about this. Have a look back through the last 6 pages or so. This table seems to get corrupted very easily with or without the usenet hack. It seems to happen on shared servers which may indicate a problem with the number and frequency of inserts and selects. However, the problem doesn't seem to happen much with the searchindex table which probably receives even more queries.

Your best bet is to empty it out and re-index. If it continues to happen then you will have no choice but to disable indexing for the usenet posts.

Do you reindex the word table through the vb cp at "Rebuild search index"? Thanks.

[webhost]

FastForward

I have installled the hack to turn the quote button into a reply button inside the newsgroups. It works fine. One question though by doing this will this also make the quote button in a regular forum other than a newsgroup act the same. It does on mind just need to know if it is suppose to or not?

Joey

[fastforward]

Originally posted by dwh


Do you reindex the word table through the vb cp at "Rebuild search index"? Thanks.
Yep.. that's correct. Select the 'Empty First' option aswell.

[fastforward]

Originally posted by webhost
FastForward

I have installled the hack to turn the quote button into a reply button inside the newsgroups. It works fine. One question though by doing this will this also make the quote button in a regular forum other than a newsgroup act the same. It does on mind just need to know if it is suppose to or not?

Joey
It works exactly as it always did in a normal forum.

All it really does differently is to pass the postid of the 'quoted' post as a form parameter back to the script.

[webhost]

Thanks . I think I might have the problem with the lastactivethread I was working on last week I forgot to put the string in the config.php file, Here is where I put it at but when I open the forum it will not open. It's torwards the bottom.

<?php
/////////////////////////////////////////////////////////////
// Please note that if you get any errors when connecting, //
// that you will need to email your host as we cannot tell //
// you what your specific values are supposed to be //
/////////////////////////////////////////////////////////////

// type of database running
// (only mysql is supported at the moment)
$dbservertype="mysql";

// hostname or ip of server
$servername="";

// username and password to log onto db server
$dbusername="";
$dbpassword="";

// name of database
$dbname="";

// allow password viewing / editing in control panel
// 0 = not visible or editable
// 1 = not visible, but can be edited
// 2 = visible and can be edited
$pwdincp=0;

// technical email address - any error messages will be emailed here
$technicalemail = "support@realwebhost.net";

// use persistant connections to the database
// 0 = don't use
// 1 = use
$usepconnect = 1;
$lastactivethread_length = 35;
?>

[fastforward]

Originally posted by webhost
Thanks . I think I might have the problem with the lastactivethread I was working on last week I forgot to put the string in the config.php file, Here is where I put it at but when I open the forum it will not open. It's torwards the bottom.

My guess is you somehow have white space at the end or beginning of the file. Also make sure you didn't save it in DOS format.

Finally, you don't actually need that in the config if you are happy with the default of 35 characters.

[dwh]

My usenet template was working fine, then I posted a message and now it's using the default template. The only other thing I did was subscribe to a usenet forum, but that didn't kill it, it was posting a message that caused the problem. I'm racking my brain but have no clue why.

[dwh]

Originally posted by fastforward

To generate a log from your cron job use something like this:

0 0,4,8,12,16,20 * * * nice -20 /home/dbforums/newnews.pl >/var/log/misc/newnews.log 2>&1

If you want the output emailed to you, just leave the redirection bits off completely. The output will be mailed to the owner of the cron job.

Uh, I always found cron complicated. what does 2>&1 do? And sorry, but can you explain the exact line for getting it emailed? Thanks!

If I didn't thank you already, thanks for an awesome script. You should get some benefit for all this hard work!

[dwh]

Originally posted by NgtCrwlr
I'm setting up all usenet forums as moderated posting only so I can keep a close eye on the spam issue.


what an excellent idea. Is this for outgoing posts? I'd like this feature for incoming posts to exclude the really bad profanity!

[dwh]

I don't understand how the spamkiller works at all. I tried to read it but it doesn't make sense to me. Can someone explain what it does and if possible to give examples like you are talking to the dummest perso there is? :( Thanks!

P.S. the latest instructions seem to still have an error?
Section 4 from vb_code_Changes
c) newreply.php - line 263

OLD CODE
--------
$DB_site->query("INSERT INTO post (postid,threadid,title,username,userid,dateline,at tachmentid,pagetext,allowsmilie,showsignature,ipad dress,iconid,visible,inreplyto) VALUES (NULL,'$threadid','".addslashes(htmlspecialchars($title))."','".addslashes(htmlspecialchars($postusername))."','$bbuserinfo[userid]','".time()."','$attachmentid','".addslashes($message)."','$allowsmilie','$signature','$ipaddress','$iconi d','$visible','$inreplyto')");

NEW CODE
--------
// START USENET HACK
$DB_site->query("INSERT INTO post (postid,threadid,title,username,userid,dateline,nn tpdateline,attachmentid,pagetext,allowsmilie,shows ignature,ipaddress,iconid,visible,inreplyto) VALUES (NULL,'$threadid','".addslashes(htmlspecialchars($title))."','".addslashes(htmlspecialchars($postusername))."','$bbuserinfo[userid]','".time()."','".time()."','$attachmentid','".addslashes($message)."','$allowsmilie','$signature','$ipaddress','$iconi d','$visible','$inreplyto')");
// END USENET HACK

----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------

how could the old code have "inreplyto" as a field?

[fastforward]

Originally posted by dwh

Uh, I always found cron complicated. what does 2>&1 do? And sorry, but can you explain the exact line for getting it emailed? Thanks!

0 0,4,8,12,16,20 * * * /home/dbforums/newnews.pl

Doing the above will email you with any output the script generates.

The redirection bit is to ensure the script output AND the cron output is redirected.

[fastforward]

Originally posted by dwh

what an excellent idea. Is this for outgoing posts? I'd like this feature for incoming posts to exclude the really bad profanity!
A simple little change to one query in newnews.pl will allow the vB moderation system to be honoured. I'll post the fix later tonight.

[fastforward]

Originally posted by dwh
I don't understand how the spamkiller works at all. I tried to read it but it doesn't make sense to me. Can someone explain what it does and if possible to give examples like you are talking to the dummest perso there is? :( Thanks!

P.S. the latest instructions seem to still have an error?
Section 4 from vb_code_Changes


how could the old code have "inreplyto" as a field?
Just a cut & paste error. I imagine there's a few more errors in the instructions too :)

As for the spam filters... unless you want to use regular expressions, you can simply enter a string leaving the defaults. If the string is contained in either the header, body or both (whatever you have selected) it will be rejected.

The option to span lines is there for safety when using regular expressions. By default, it will only match if the string is all on one line. Without this check it would be very easy to inadvertently match every message and reject the lot. It's impossible to give you a tutorial on regular expressions in one post. I recommend you do a quick search and put aside a few hours, weeks or days :) to learn all the possibilities. The script uses standard Perl regular expresions (as opposed to POSIX used by ereg in PHP).

I'll post some examples when I get home later.

[dwh]

That helps a bit, and I'll live w/o learning the regexes :)

What I still find confusing is the following options:

Delete vs Edit, plus select button?? Delete what? Edit what?
-----

then lower

Add vs Update, plus submit button.

Are those 2 separate forms, hence separate submit buttons? If so an <HR> tag would be nice.

----
BTW, this is only meant as constructive suggestions but you went WAY above the call of duty, how you say?

[fastforward]

Originally posted by dwh
That helps a bit, and I'll live w/o learning the regexes :)

What I still find confusing is the following options:

Delete vs Edit, plus select button?? Delete what? Edit what?
-----

then lower

Add vs Update, plus submit button.

Are those 2 separate forms, hence separate submit buttons? If so an <HR> tag would be nice.

----
BTW, this is only meant as constructive suggestions but you went WAY above the call of duty, how you say?
The [ Select ] button places the higlighted line into the edit fields when the 'edit' radio button is selected, or deletes the highlighted line when the 'delete' radio button is selected.

The submit works the same way, but you get to chose whether you want to add whatever's in the edit boxes as a new record, or edit the existing one.

I tried it with seperate forms and it was a lot more confusing, trust me. Although they are seperate forms, once the first [select] button is pressed, the other button comes into play for the pre-filled edit fields on the same form. Besides, the usenet.php file is an undocumented mess and there's no way I'm going to back to re-write it :) . I will fix the [Review/Insert Orphans] option though... one day.

[dwh]

well my custom usenet template still doesn't work, maybe I'll try deleting it and redoing it. But I forgot to write down which templates need modifying. I think it was postbits and also newreply for the quoting....

In the meantime, it looks like when you put in a smilie, the post shows up twice, once with the smilie translated and once with the smilie as plain text...

[dwh]

OK, fixed the template problem. Turns out even if you don't allow users to choose their customer templates, when choosing a template you still have to click on "override user templates for this forum".

One more technical question. I'll give an example posting.

This is vb style linear forum
---
1. Post 1 = test1.
2. Reply1 to Post1 = hello.
3. Reply2 to Post1 = hello2.
4. Reply1 to Reply2 = hello3.
5. Reply1 to Reply1 = hello4.

In a threaded forum it would appear thusly:
---
1. Post 1 = test1.
2. Reply1 to Post1 = hello.
5. Reply1 to Reply1 = hello4.
3. Reply2 to Post1 = hello2.
4. Reply1 to Reply2 = hello3.

Since usenet is threaded, it should up the second way, right? And this seems to be the case. But not when I'm sending a reply via my forum.

Do I need to modify the postbit quote button, or the "Reply" button to point to something other than the first topic? Oh, while writing this I realize that there needs to be a reply button on each topic for this to work right, so I guess it won't. Never mind.

[fastforward]

Originally posted by dwh
OK, fixed the template problem. Turns out even if you don't allow users to choose their customer templates, when choosing a template you still have to click on "override user templates for this forum".

One more technical question. I'll give an example posting.

This is vb style linear forum
---
1. Post 1 = test1.
2. Reply1 to Post1 = hello.
3. Reply2 to Post1 = hello2.
4. Reply1 to Reply2 = hello3.
5. Reply1 to Reply1 = hello4.

In a threaded forum it would appear thusly:
---
1. Post 1 = test1.
2. Reply1 to Post1 = hello.
5. Reply1 to Reply1 = hello4.
3. Reply2 to Post1 = hello2.
4. Reply1 to Reply2 = hello3.


Since usenet is threaded, it should up the second way, right? And this seems to be the case. But not when I'm sending a reply via my forum.

Do I need to modify the postbit quote button, or the "Reply" button to point to something other than the first topic? Oh, while writing this I realize that there needs to be a reply button on each topic for this to work right, so I guess it won't. Never mind.
Assuming you added the $postid hidden variable to the template, that's all you have to do.

The behaviour you are seeing shouldn't happen very frequently in normal use. My guess is that you posted a bund of replies between usenet pulls. Remember, the msgid is not allocated until the post reaches usenet. Then we have to wait for the next pull to actually find out what it was allocated. This can cause slight ordering problems if a lot of local posts are made in succession in the same usenet forum.

[dwh]

interesting! I see, ok....

I'm almost done but what's missing still is that outgoing messages don't always go out. So if I get you right, it's waiting for the message id to be assigned and only then will it go out? Cool!

[fastforward]

Originally posted by dwh
interesting! I see, ok....

I'm almost done but what's missing still is that outgoing messages don't always go out. So if I get you right, it's waiting for the message id to be assigned and only then will it go out? Cool!
Yep... that's exactly right.

Now you've got it all installed and working, I'm about to release a new version, so you'll have to do it all again :D

I'll have it ready tomorrow. Some changes are:
the support for moderation
a new module is used to handle email addresses (Mail::Address) Replies are handled differently. It is now far less strict about having an immediate prior message number before being inserted. It should get rid of the large number of orhans.
A lot of other little changes aswell.

[george_proost]

when checking the words table this message appears in the log.


DBD::mysql::db do failed: Duplicate entry 'cheesehead!' for key 2 at www/admin/newnews.pl line 628.

can it be suppressed ?

================================
Also I seem to be getting a loop

...
...
loading 1 replies...
loading 1 replies...
loading 1 replies...
...
...

[webhost]

Great news.
Thanks Fastforward

3 cheers to the orphan Eliminator

Joey

[george_proost]

when does it appeear ?

hack on a hack:

I duplicated the newposts() code in search.php called it
newpostsusenet() change the SQL select and bingo ..

You can then offer new forum posts and new USENET posts search links.

Can this be included in the package please?

[fastforward]

Originally posted by george_proost
when does it appeear ?

hack on a hack:

I duplicated the newposts() code in search.php called it
newpostsusenet() change the SQL select and bingo ..

You can then offer new forum posts and new USENET posts search links.

Can this be included in the package please?
Good idea... I never thought of doing it this way. Simple and effective. Just the way I like it :D

I hoped to get the new version out today, but it's already 11:30pm and I haven't documented the code changes or SQL changes yet. I'm afraid it will have to be delayed again. Sorry. I'll see if I have time at work tomorrow to finish it off.

[shri]

Fastforward,

Could you PLEASE put the new version in a new thread? :)

Cheers,
Shri

[george_proost]

A USENET button

https://vborg.vbsupport.ru/

[andy_C]

Great news on the new release.


Has anyone managed to get a free outgoing message server working? I'm trying to use pubnews.netcom.net.uk, albeit without luck. It says that it is working, (posting message to whatever.newsgroup by whoever) but, alas, it isn't.


I've spoken to newscene, and may buy an account with them - not using too much transfer - but I'd prefer to keep costs down if if can.

Has anyone managed to get it working? Is it a fault in my installation or does http://pubnews.netcom.net.uk not work? Is it worth getting a separate, paid account?

Many thanks.

[webhost]

Andy I use usenetserver.com. about 20.00 every 3 months, 57,000 groups great upload and download time. A very knowlegde guy told me about them from this board.
I tried the free ones not very reliable.
Joey

[andy_C]

That sounds great. Did you have a chat with them beforehand about their terms of service? Or is the existing terms of service cool with using it for a gateway?

Thanks for your help.

[webhost]

I talked with them on the phone and told them what I would be doing and how much I would be polling them. They replied no problem.

[hhdream]

DBD::mysql::db do failed: Duplicate entry '[/size' for key 2 at newnews.pl line 626.
DBD::mysql::db do failed: Duplicate entry '[size=1' for key 2 at newnews.pl line 626.

Could you please answer - why it appears only when usenet script runs?

[andy_C]

try turning off indexing. worked for me.

[annams]

No, pubnews.netcom.net.uk did not work for me either. In my case too, the program said the postings are sent.

I wonder if this is a temporary problem!

Originally posted by andy_C
Great news on the new release.


Has anyone managed to get a free outgoing message server working? I'm trying to use pubnews.netcom.net.uk, albeit without luck. It says that it is working, (posting message to whatever.newsgroup by whoever) but, alas, it isn't.


I've spoken to newscene, and may buy an account with them - not using too much transfer - but I'd prefer to keep costs down if if can.

Has anyone managed to get it working? Is it a fault in my installation or does http://pubnews.netcom.net.uk not work? Is it worth getting a separate, paid account?

Many thanks.

[webhost]

Save yourself some headaches and get a pay server it is the best. I tried a couple of free ones up front and had problems with all of them since I changed to a pay server. No more problems!

[fastforward]

Originally posted by shri
Fastforward,

Could you PLEASE put the new version in a new thread? :)

Cheers,
Shri
I think there have been so many changes to the script since this thread was started that the early messages are becoming irrelevant, so I think you're right.

I will start a new thread with the next release :)

[george_proost]

I have orphan problems galore... please please

even a beta-pre-release-trial-demo-version

[fastforward]

THIS THREAD IS CLOSED!
Posts in here will NOT be answered

This hack has undergone many major changes since this thread was started. Consequently, most of the posts have become dated and of little use. To coincide with the latest major release (20010712), a new one has been started.

See this thread (http://www.vbulletin.com/forum/showthread.php?s=&threadid=22599) for the latest version and discussion

[dwh]

Originally posted by andy_C
try turning off indexing. worked for me.

Nice picture, you look familiar. :)

[andy_C]

Originally posted by dwh


Nice picture, you look familiar. :)

i get around.:cool:

[robertusss]

Originally posted by webhost
Save yourself some headaches and get a pay server it is the best. I tried a couple of free ones up front and had problems with all of them since I changed to a pay server. No more problems!

since this is closed, maybe we can use THIS thread to discuss news-servers...

you you please share with us what paid-server you use?

thanks