heh get on msn gilby mate.. time for a little chat ^_^

what about that pop3 file in version 1.9?

do I need to upload that at all?

Yes, that's for the email capabilities.

Same database. Just go in to that database under phpMyAdmin and there will be a box wher you can type SQL to execute, and underneith that is a file browse input where you can select an sql file to execute. You can run it from either, the file way is easier.

Post 1 of 2

Gilby,

Attached is a image of a linking problem I was on about on MSN, with regards to the RE: posts not joingin together under the email to forums addon part of this hack.

You asked me to add [FFML] to the prefix section on the MySQL database entry which I did, see the image in post 2 of 2.

Any ideas?

Post 2 of 2

Hi,

getting problems if I do a 'Update Thread Info - update thread post counts, original posters, last post date, attachment totals' command via the Up-Date Counters section on the admin cp.

This only started to occur then I updated to Gilbys 1.9 version of the NNTP/Email feed, I belive all the newsgroup based forums (forumid)s are affected, I cant tell cos it bombs out on the first newfeed based forum it tries to update the counters on.


The error given is:

Forum ids:

Processing forum 89

Warning: MySQL: Unable to save result set in /usr/home/wocsnet/public_html/admin/db_mysql.php on line 93



There seems to have been a slight problem with the Sanjiyan's Japanese Anime and Manga Forums database.
Please try again by pressing the refresh button in your browser.

An E-Mail has been dispatched to our Technical Staff, who you can also contact if the problem persists.

We apologise for any inconvenience.

Database error in vBulletin Control Panel 2.2.5:

Invalid SQL: SELECT lastpost,lastposter FROM thread WHERE lastpost='1017967998'
mysql error: Got error 127 from table handler

mysql error number: 1030

Date: Friday 05th of April 2002 02:45:19 AM
Script: http://www.wocs.net/admin/misc.php
Referer: http://wocs.net/admin/misc.php?action=chooser


Also the forum it fails on as well as being NNTP based, is also the email forum, ie the forum that deals with and posts the POP3 mailing list feed.

I've got this hack working and love it!! Thanks!

Question to everyone, though. What kind of speeds is everyone getting from their servers. Reason I ask, is I log in via OE to my news server and get great throughput, but when I run the gateway.php I get speeds in the 500 bytes/sec area. Big difference... Could anyone have any thoughts as to the speed difference. The only thing I can think is my host has the speed throttled for incoming connections or something.

Thanks!

Take out the brackets, that's implied in the prefix column.

And an upgrade to 2.2.5? do you know if it happened with 2.2.4 of vb and 1.9 of the gateway?

didnt use 1.9, I used the version you edited for me.

So 2.2.5 and 1.9 are both new.

> didnt use 1.9, I used the version you edited for me.

What version did I edit for you? If I did, it was from a while back and
would not have the email support.

- Gilby

nope it didnt have email support, it was the version that you made for me that reported the feedback.. it was a while back ^_^

Has anyone tried the "prune messages" function with this? It seemed like the first time I did this, it reset my 'last msg ID' field to 0?

I absolutely *love* this hack and would dearly like to use it on my board. However the long usernames from usenet tend to be cumbersome indeed. Is there a way to hack gateway.php so that let's say only the first 20 characters of the username are uploaded? I am not really worried about chopped off usernames.

I agree, this would be a nice addition.

P.S. Gilby this is my #1 favorite hack (and I have over 25 of em installed, and have created about 15 of my own) thanks again! :)

I figured it out myself and it seems to work OK. Ideally I would like to see the username length as a parameter in the nntp_settings table :)

In gateway.php find each instance of:

and replace by

to get a 15 character delimited username.

Yes that happens to me to. I guess you can run setlastmsg.php again as a workaround but it would be nice if the last msg ID was preserved automatically.

Better even would be if the NNTP gateway had its own pruning mechanism for those forums in which you import the usenet posts. For e.g. if you schedule the gateway job every hour, it would be *perfect* if it could also check for posts older than x days and remove them from the board :)

Gilby, would you consider such an addition?

Are you using the prune messages function that is in the vb control panel? I can't see how or why that would even touch the nntp table in the database.

Gilby, yes I am (also) using 'Mass Prune' from the AdminCP and the behaviour is consistently the same. Deleting all posts from a forum resets the msgid field. Btw, I am using vb 2.2.4

Superb hack though :) I hope you will continue developing it.

Regards,

Pat.

Hmmm... I just tried using the mass prune (the one built into vb), and it didn't change my lastmsg field in the database. I'm confused on why this would even happen though. vB doesn't even know the nntp tables exist when it executes that. unless you are using some other prune script.

Is this mass prune in the nntp graphical interface? If so, then that'll explain it. I don't have that hack installed. In the admin control panel, the 'mass prune' found under the 'Threads & Posts' heading did not change the lastmsg field for me.

Working smoothly with 2.2.5, if anyone was wondering. :)

Gilby, it seems weird to me too but yes I am using the standard vbulletin 'Mass Prune' option to delete all posts. I don't have the graphical NNTP admin panel installed. But I did some more tests and apparently the problem does not appear when I only prune the full contents on 1 single forum. Previously I had selected the parent forum including subforums. I will run some more tests if I get the chance :)

Using older stuff from Usenet hack - you need a new field in thread table.

ALTER TABLE thread ADD isusenetpost TINYINT(2) DEFAULT 0 not null;

then open your gateway.php and find:

// Create thread
$DB_site->query("INSERT INTO thread (threadid,title,lastpost,forumid,open,replycount,p ostusername,postuserid,lastposter,dateline,iconid, visible,attach,msgid,prefix) VALUES (NULL,'".addslashes(htmlspecialchars($subject))."' ,'".$date."','$forumid','1','0','".addslashes(from _name($message[from]))."','0','".addslashes(from_name($message[from]))."','".$date."','0','1','$attachcount','".addsla shes($message[msgid])."','".addslashes($theprefix)."')");

replace it with (changes enbolded)

// Create thread modded
$DB_site->query("INSERT INTO thread (threadid,title,lastpost,forumid,open,replycount,p ostusername,postuserid,lastposter,dateline,iconid, visible,attach,msgid,isusenetpost,prefix) VALUES (NULL,'".addslashes(htmlspecialchars($subject))."' ,'".$date."','$forumid','1','0','".addslashes(from _name($message[from]))."','0','".addslashes(from_name($message[from]))."','".$date."','0','1','$attachcount','".addsla shes($message[msgid])."','1','".addslashes($theprefix)."')");

From now on each thread created by gateway.php will be flagged as originating from usenet (isusenetpost=1)...

Open search.php and find two lines:

$wheresql.=" AND thread.open<>10";

in my case around line 551 and 601...
Directly below each of them add:

// NNTP Gateway mod
if (!$includeusenetpostsingetnew) { $wheresql.=" AND thread.isusenetpost=0"; };
// End of mod

Having isusenetpost in thread table also allows you to separate threrad counts or create separate links for "View new Usenet posts" ( see example here...

I had the same idea but implemented it differently, without the added table field:

Open search.php and find:

After that, add:

Then update your forumhome_newposts template to include 2 links rather than one:

To look for normal posts:
<a href="search.php?s=$session[sessionhash]&action=getnew&usenet=false">

To look for usenet posts:
<a href="search.php?s=$session[sessionhash]&action=getnew&usenet=true">

Not being a PHP wizard I am not 100% sure that this would cover all angles but it seems to work for me fine so far :)

I'm not a server admin, and the server is set on safe mode. Which is really pissing me off. Also I have to use phpMyAdmin since they have Telnet disabled. Any help on how to reorganize tables and such would be appreciated.

Hey, Gilby... Thanks for the awesome hack!! =)

I got a question. I see you fixed the mime decode problem in the subject and from fields. I pruned my usenet groups, recompiled php with imap support, and re-imported everything... and sure enough, the froms and subjects were fixed.

There's still some mime in the text, though, that goes away. Especially when some of the header leaks into the text of the message.

Anyway to apply the same decoding to the body of the messages, too?

Oh... I have another question.

Some of the groups I mirror have this one single harassing user who posts something like 20-50 messages a day. I'd like to somehow filter out his messages. I feel that it could make these groups usable again.

I tried pruning by username. Nothing came up... so I then added his various user names as different users on my system. THEN I could prune by username. What then happened was half good. In prune by username, threads he started showed up, but no posts.

Any idea how to filter out the bad seeds? The hacky idea I have above would work if only imported posts, not just threads, could somehow be attached to a username. I know vb.org accomplished this somehow. They grabbed the hacks forums from vbulletin.com, then when people registered here, it would attach their user name to the name they used on their old messages, and whala... full vb user based functions could be applied to old messages.

Gilby... Thanks for the great hack!!

I would like to exclude Usenetposts which contain a x-archive = no in their header - has anyone already done this? or is there a hint at which part of the script i can sort those posts out?
Thank you!
Tom

I've battled with this hack for the best part of the last 3 hours and I've given up in total frustration. :tired:

No matter *what* I do, all I can get out of gateway.php is this:

1 group(s) gatewayed.
Logging in to news.usenet.com, group alt.sports.airsoft

nothing else.

This is what I have in nntp_groups:

alt.sports.airsoft
<nothing>
98
0
news.usenet.com
1003989135
<password>
1

Help :cry:

Try the attached nntp.php file in place of the one you are using. this will output the responses from the nntp server.

Look at the statement that checks if it is a forum post that is being read from teh newsgroup (if ($message[useragent] != $settings[useragent]). Add an exclusion for that header too. You may need to modify something else so that the header shows up in the message array.

A simple SQL query can be used periodically to delete them. Use something like "DELETE FROM post WHERE username='name of bad seed'"

You'll also need to run DELETE FROM thread WHERE postusername='name of bad seed'"

Or you can see my last post and add another thing to the if statement.

ok, thanks for that Gilby. I've got a bit more information. It logs in ok, but then does nothing? I tried it with uk.test last night and got some action, but there are posts in alt.sport.airsoft (not many granted), but still some. I'd have thought it would at least import those few? Here's what I got:

1 group(s) gatewayed.
200 Welcome to Usenet.com, 5.5g6, S8
381 PASS required
281 Ok
211 6 621 626 alt.sport.airsoft selected
Logging in to news.usenet.com, group alt.sport.airsoft

I think that vB deletes threads by username, even usersnames used in usenet posts. But the SQL query you gave me to delete posts by username... wow, that's useful!

So is the filter heading instructions. Great info, thanks!!

BTW... I've found two possible issues...

The first is posting to Supernews. I couldn't get it to post WITHOUT removing "Message-ID: <$msgid>\r\n" from the $response string. Supernews always ACCEPTED the messages just fine (240), but they just weren't showing up. I guess that either they have a beef with the way you generate the msgid OR they prefer to generate their own, and their server gets "insulted" by the fact you're offering your own. I don't know.

This didn't happen using Giganews, incidentally. Everything posted there just fine w/o modification. I only moved to Supernews a couple days ago, and still have my Giga account 'till the end of the month in fact. I believe it always worked well with Easynews (they have a $1 trial I checked out).

Side note to others reading -- Those services I just mentioned are probably the "big 3" of services offering quality feeds with high rentention, in case anybody is interested. If you want something cheaper but still good ($5mo), check out airnews.net. SUPER cheap (free) but still quality is readfreenews.net.

Reading access is free and needs no password. To get posting access to readfreenews, net, you have to email drechsau+usenet@yuck.net (Mike Horwath), with real name, real email isp email address, username wanted, password wanted.


Alright... next possible bug...

Sometimes when pulling groups that have an a large number of messages.. generally 25-80 thousand... the httpd process (with php as a module) begins swelling in memory access VERY VERY rapidly. Say 2mb growth every few seconds. After 1000 messages, I might have a 400mb process! It will then proceed to fill up the whole of available memory AND swap. But it'll still work. After the first time this happened, I've been killing these processes gone wild, though.

Sometimes this doesn't happen. Sometimes I can tell it to grab 40k messages or whatever... and see perfectly ordinary memory usage... a growth of just a few k every few seconds. Normally, it ends up being something like a 130mb or so after tends of thousands of message havebeen gotten, not just a few hundred.

This tends to happen more AFTER a reboot or a successul restart of Apache. What this seems to indicate to me is that some sort of memory routine isn't being initialized properly... but after php and apache have been used for awhile... everything that could be initialized in apache has been... That's just my theory, I could be wrong.

No matter what it is, though, this might be more of a php issue (i'm using the latest non-beta self-compiled version up on the php.net site... 4.1.2) than a problem with your script. Since it's memory related, I even thought it might be kernel related, but I just updated quite a few revisions and it still happens. I even recompiled php with different options. Same problem. Guess I'll have to wait for the stable release of 4.2 to see if it still happens.

Alrighty... thanks a bunch for your help and continued attention, Gilby! This is a great hack. You should really be compensated for this. Consider having some sort of donation system that parallels vbportals. I say vb portals, because the combination of everything they do to get donations is very successful. Offering betas only if you've donated and the other things they do really does seem to get them more donations than normal.

Is it at all possible that, when a local (vB) reply to a newsgroup is deleted, it gets cancelled off the newsgroup as well?

Just a thought :)

Installed ok but Im getting the following error:

<br>
<b>Warning</b>: Failed opening 'class.POP3.php3' for inclusion (include_path='.:/php/includes:/usr/share/php') in <b>/home/virtual/site1/fst/var/www/html/forum/gateway.php</b> on line <b>25</b><br>
<br>
<b>Warning</b>: Cannot add header information - headers already sent by (output started at /home/virtual/site1/fst/var/www/html/forum/gateway.php:25) in <b>/home/virtual/site1/fst/var/www/html/forum/gateway.php</b> on line <b>483</b><br>
0 group(s) gatewayed.

Any help?

Brian

Uploaded class.POP3.php3 to the forums dir and now no errors. But all Im getting is this:

0 group(s) gatewayed.

and no posts from newsgroup in the section?

Most news clients generate their own message ids so I'd guess that there is something with the format that the message id is in. I'm not aware on if there is a standard format for the message id, just that it needs to be unique, but I'll look into it to see if there is one that I was not aware of. It could also be that the supernews server doesn't accept a message id that has the host part of it different from where it's being sent from, so you may want to look at the posts that are made to it and see what the message id looks like. Then use that host for the setting in the nntp gateway.
Hmmm... I hope it's a problem with something other than my code. ;) However, I'll take a look at this when I get a chance and maybe try out some large imports to see if I can duplicate this.

Meanwhile, you can set a limit of how many posts you want per pull per newsgroup by putting the following code before the line that contains (for ($current = $lastmsg+1; $current <= $max; $current++)), about line 90:
If it's a problem with my code, then php will free the memory when it's done executing the script (it may also do this is the problem is elsewhere).

I've been thinking of possibly calling it "donationware" which simply means if it's useful for you and you can afford to contribute some cash, then please donate. So, if you want to donate, you can use this PayPal link: https://www.paypal.com/xclick/busine...r=nntp-gateway

Did you set up a newsgroup to import? In the nntp_groups table, you need to add a row that has the server, newsgroup, username, password, etc for the newsgroup you want to gateway in. If you don't know how to add a row in the mysql table, you may want to install the GUI that was made for this hack: https://vborg.vbsupport.ru/showthrea...threadid=36475