Hello

I think the script works much better if it's run using plain cron, not vB scheduler. If i run it through vB scheduler and ouput some variables inside it they are NULL or have strange values.

Also i think one of the problems is not unique msgids. Now i'm using seconds since epoch + microseconds + user name and it helps.

Hope it will help someone

Just remove posting privileges in the forums where you don't want users to post :)

Is anyone else having problems with A Small Orange and Giganews? I got an email from someone else saying their Usenet forums stopped receving posts so I checked mine - nothing new since May 21.

ASO is fixed. They stood up a new firewall and forgot to open port 119 ;)

I assume I am thinking this hack is something other then what it is. I set it to grab group alt.binares.dvdr well that was a mistake. Is there a a way to get it to grab just the headers or something not the whole attachment that is just a bunch of garbage in the post casue it showed up like ASADJJKMSDA><MDSNDLSDJMD>SAD witch i feel sure is the rar file just not as a atachment. I guess basicly I was thinking it would something like newzbin where it just showed the headers. ( altho would be nice if you could make it so it would make a nzb file to download. ) Do I have this hack all wrong in my head?

Sorry if this has been asked already. I am in the process of setting my site up and importing a lot of newgroup posts. Every once in a while it will just stop for no reason and give no error at all. Then I can change the last run time and start it again.

Any idea what this could be?

Is there a way to support one mailing list email (eg: 1 Yahoo group) going to different forums via the prefix option? Can I remove the database uniqueness restriction on this field (NEWSGROUP in NNTP_GROUPS table) or are there any other ways around this ???

TIA -Xuan

How do I make it look for new post. Also do I have to have the last message ID if I am just starting this hack up for the first time?

The time the script will execute before timing out is controlled by your host in the php.ini file.

If you have a low timeout time it can be a pain doing the first import, you could always use the crontab and leave it for a day to do itself.

It will only look for new posts if you put it on crontab or manually run it via the gateway file.

You don't need a last message ID for the first run, it'll fill that in automatically after.

I see the recommended cron job is :

lynx -source http://domain/forumdir/gateway.php

Is there any reason we can not just set our cron job to :

php /pathtogateway.php/gateway.php

?


Also has anyone had any success running multiple instances of gateway.php for multiple forum installations on the same server? Any problems?

Last but not least, is their a recommended maximum amount of newsgroups that we should do per forum instance or will gateway.php handle whatever you got?

I run 3 instances of gateway.php running on 3 different forums and don't have any problems.

How did you do this? I would also like to run php 5 but can't seem to figure out how to make gateway.php to run 4.3.11

I run Windows and IIS so I just set IIS to use php5 isapi filter and for scheduled tasks and command line php 4.3.11 cgi is used. For a while I was using .php and .phtml extensions as php5 and .php4 for php4. Have to add a registry entry for php5 to point to php.ini and for php4 set the user path.
I don't know how you'd do it on Linux, but, it can be done.

For Unix you can set up a cron job and use the 4.3.11 PHP CGI program to run gateway.php.
eg: this will run every 10 mins
1,11,21,31,41,51 * * * * /path-to-php4.3.11/php-4.3.11 -f /path-to-script/gateway.php

BTW, I am trying to setup a new gateway for a forum and I only want futures postings to be forwarded to a mailing list, not old posts. How could I do this? Can I set the Last Max postid # to the last posting # of that forum??? TIA

Yeah I got that all working, but now I have a problem where the script is not following the its own time constraints. From what I understand it is supposed to run no longer than 30 minutes. Also another instance should not run within 30 minutes of the first starting. This makes sense so that you dont have multiple copies running at the same time.

My problem is that gateway.php seems to be running for 2 hours + so that if I cron it multiple instances will occur. Anyone else run into this problem?

I took a backup of my vb database to my home linux box to test out NNTP gateway to a Yahoo group mailing list. I only enable 1 sub forum and left the default value of 0 on the Last Max postid field. For the first initial run, the gateway sent ALL messages from ALL forums including those msg from an un-related PRIVATE forums to the Yahoo list??? Luckily! this is just a test else I could be in big trouble. I believe this problem only occurs in the initial run.

I just switch NNTP on for the real server with appropriate Last MAx postid set to the last posting #. However, I ran into 2 issues:

• I do not see any messages arriving at Yahoo. When using with debug option, it appears that messages are sent ??? BTW, Importing messages from a POP3 account works.
• when running the gateway.php script via PHP CGI (command line), I encountered an error message: 'Unable to add cookies, header already sent.'

The only different is: my hosting company uses PHP 4.3.10 whereas I have 4.3.11 at home. Perhaps there are some other restrictions?? I use 1and1.com btw.

Any idea of the modifications that need to be made to get this working with 3.5 beta?

I am going to compile php as cgi mode just for this script what do i need to include to make it run?

<b>Fatal error</b>: Call to undefined function globalize() in <b>/home/usenetma/public_html/gateway.php</b> on line <b>49</b><br />

Vbulletin 3.5 beta 2 error message.....

I'll pay $50 to the first person who creates a VB 3.5 compatible version of this... I'm sure some others would be willing to chip in too

I'm working on the mods to make this work with 3.5 and I've found where the post table has changed a bit so I'm working on getting things working right with that...

All DB queries have changed and the globalize has changed..... so it is a decent amount of work to get this working

it didn't work it just output the html to the display

I got parts of the import functioning on 3.5... it appears to grab the text but attatchments and debugging output aren't working and I haven't had the time to do any further testing.

The main thing I need for my new project is the inline mod tools and attatchments... I've decided to stick with 3.0.7 and do the inline mod hack and stay there.

ah well :(

Is there a way to have the username display as something other than "Guest" to something like "UseNet User" instead when posts are imported in from the newsgroups. Without changing the default guest account on VB. I know this sounds trivial but I have actually had people complain that it makes them look like they posted from the site rather than the usenet group.

Tremendous mod :)

Only problem is the speed. I seem to be getting only about 3,500 posts/hour, whereas it takes under a minute to download that number through NNTP on a newsreader. But I suppose that's probably because of the vBulletin database functions taking up a lot of time, rather than the download of articles itself...?

Anyway, a very helpful mod, and much appreciated.

I see some people here having problems posting with Giganews; not being able to upload articles to Usenet (error code 440). If you have just signed up with them, this is because of the free trial period?during the free trial period, you can download articles, but you can't post; they say it's because of preventing spam. You have to manually bypass the trial period in order to be able to post right away.

kadence, the biggest issue you'll have with the speed is how fast your hard drives are. Inserting loads of data will take fast drives housing your MySQL servers. I suggest at least 10k SCSI drives with 15k obviously being faster.

I believe you must be using PHP <= 4.3.10. In this case, just take out the -f option and use something like:

1,11,21,31,41,51 * * * * cd $HOME/forum; php gateway.php > $HOME/tmp/null

After a lot more work than I expected :speechless:, I was able to modify this so that it works with 3.5 Beta 3.

What I did:

• Replace $DB_site with $db in gateway.php, nntp_groups.php, nntp_settings.php, and functions_nntp.php.
• In those same files, replace all instances of globalize( with $vbulletin->input->clean_array_gpc( ; and the tricky part: alter $_GET, $_POST, and $_REQUEST parameter inputs within clean_array_gpc to 'g', 'p', and 'r' respectively; and change the variable calls from $variablename to $vbulletin->GPC['variablename'].

Note that altering nntp_groups.php isn't entirely necessary, as the nttp_groups table can be directly modified via phpMyAdmin.

In nntp_settings.php the affected lines (in terms of the original line numbers) are around lines 92, 107, 141, 192, and 223. In nntp_groups.php the lines are around 324, 336, 368, 414, and 451.

Rather than replace all variable instances with $vbulletin->GPC['...'] calls, I just set new variables with the same names as the old ones, using $vBulletin->GPC['...']. e.g., add the line $ngid = $vBulletin->GPC['ngid'] right after the clean_array_gpc call; the variable $ngid can then be used the same as before.

One thing that gave me a bit of trouble was line 241 (in my modified file, line 263) of nntp_groups.php. In this line $group[?forum?] must be changed to ?forum?. Otherwise the form variable name comes up as ?0? on the post form, rather than ?forum?; and thus the table update doesn't work.

Note that I made a change to gateway.php line 71, and I'm not sure if it was complete; this would affect the 'log' and 'debug' variables, but those don't seem to be used anywhere that I could find.

The 4 files that were changed (nntp_groups.php, nntp_settings.php, gateway.php, and functions_nntp.php) are attached in .zip format. Note that I did not test thoroughly, so "Use at your own risk" warnings apply!

To add a menu item in the admin control panel, copy the file "cpnav_nntp_gateway.xml" to the /includes/xml directory.

Ah, duh?$debug is used when passing ?debug=1 in from the URL ;) Without it the logging() function doesn't work.

So I added the following lines after line 71 in gateway.php:
The .zip file in the post above has been updated.

Thanks much. I'll give it a try on my test box.

can someone help me here.... i have most of it all installed but the file tries to download...so i went to the ms site and attempted to edit the registry. i found the key and added a new value and got it all up to the point of adding the value data. It wont let me add HEX 0x1. Id really like to install this and get it all working...thanks in advance.

Key: HKEY_CURRENT_USER\Software\Microsoft\Windows\Curre ntVersion\Internet Settings

Value name: IsTextPlainHonored
Value type: DWORD
Value data: HEX 0x1

FIXED IT....WELL ACTUALLY RAN THE FILE ON ANOTHER MACHINE

OK?I made some more changes :)

There was an issue with form variable names on nntp_groups.php line 278 (original line 256), of the same variety as the print_forum_chooser problem on line 263 (241). This was causing problems with "Add Newsgroup", rather than "Edit"; when trying to add a newsgroup, the forum would be set to 0, rather than the forumid.

Also, I replaced all instances of $vboptions['...'] with $vbulletin->options['...'] and global $vboptions with global $vbulletin in functions_nntp.php. Looks like this affects a lot of attachment stuff.

I have no idea if attachments work now; perhaps someone will test it and share their results :squareeyed:

My earlier post has the updated .zip file.

Would it be possible to use this to share posts between sites? How would it be done?

vB indexes all the words in the post contents, that is why it is very fast to search a term from the post. Whereas a normal news reader usually only index the subject. To find a keyword within the post would basically require a full article scan. I believe the indexing takes up most of the time for importing.

To speed up a bit, one can delete the index for 'msgid' in the post table, and reindex that field with a smaller size. It is quite easy to do with myphpadmin. Just set the size of the index to 10 or 15. That might(according to mysql manual) improve some speed.

I actually mentioned this somewhere, if not in the instruction, it will be somewhere in this thread.

The script tries to check if there is another instance of script running within 30 minutes from the time the last script was run. It will ignore the checking if it is over 30 minutes and considers the last run crashed.

So it is important not to set up the cron job before the completion of the initial import, as the initial importation may take too much time.

Could you give us some more information on this as well as anything else that can speed the import process up... So far I've just been tossing SCSI drives at it and simply waiting LOL

To delete the old index:

ALTER TABLE `post` DROP INDEX `msgid`;

and to add a partial index for msgid:

ALTER TABLE `post` ADD INDEX ( `msgid` ( 10 ) ) ;

10 means instead of indexing the whole length of the field (128), we only index the first 10 charactors. This will reduce the size of the index. The full length index is fastest at the time of retrieving, a partial index means the DB engine needs to read more lines at the time of retrieving but needs less work at writing.

To people who needs to import large amount of posts in one go. I think if you disable the index building process at the time of importing, it will speed up a lot. You can then rebuild the index using the AdminCP. Bear in mind, rebuilding the index takes a VERY long time. The total time used to rebuild the index should be less than the time used to import large amount of messages and build the index at the same time. I have not done this, this is only base on the theory that it is very slow to insert records into a fulltext index table, and it is faster to build the fulltext index after all the insertions are done.

If you do not need to search the imported messages, then disabling the index build will surely speed up the import. To do this, just comment out

build_post_index($postid, $foruminfo);

line in the /include/functions_nntp.php

I'm gonna go ahead and test the 3.5 version on my test version. I'll let you all know how things go.

the 3.5 version works good an does support attachments from the testing I've done....