vb.org Archive - NNTP Gateway for Usenet ( Newsgroups ), Mailing Lists

vb.org Archive (https://vborg.vbsupport.ru/index.php)

- vBulletin 3.5 Add-ons (https://vborg.vbsupport.ru/forumdisplay.php?f=113)

- - NNTP Gateway for Usenet ( Newsgroups ), Mailing Lists (https://vborg.vbsupport.ru/showthread.php?t=92588)

Quote:

Originally Posted by itdevil (Post 1366623)

I am having similar issues, the mod put too much load on the server, i am 2x DualCore Xeon 2.8 with 2GB ram 100mbit.... Cron set to run every 24hrs or midnight :o... once the cron run the load shoots up to 6 - 20 till apache crashes....

even now the server is dead slow due the gateway.php cron

Code:

xyz 18394 11.2 3.6 86660 74972 ? Ss 00:00 25:39 /usr/bin/php /home/xyz/public_html/forum/gateway.php

i have also noticed, sometimes gateway file (but stop leeching from news server) never get killed automatically, on next cron another process is created...

This is a great hack, but someone need to update and optimize the code.

I haven't had the same problem, but I would recommend running it more than once a day. I run it every 10 minutes. Otherwise you'll have to download a significantly more amount of data

the largest bottlekneck in large scale archiving is the hard drive speed. If you don't have high speed SCSI drives don't ever try to archive millions of posts! Run things manually and NOT through cron if you're weeks behind in the posts.

i build a new nttp gateway from zero using php, C++ and Delphi. it's works very more efficiently than the actual nntp gateway (i test it on a 100.000 articles newsgroup without any problem) and also i inprouve the charset encoding/decoding and correct some litle bug i found in gateway.php

exemple : after inserting an article
$db->query("
UPDATE " . TABLE_PREFIX . "thread
SET firstpostid = $postid
WHERE threadid = $threadid
");

that look false and must be

$db->query("
UPDATE " . TABLE_PREFIX . "thread
SET lastpostid = $postid
WHERE threadid = $threadid
");

:)

i need now some volonters to test it and to gave me they comments !

Quote:

Originally Posted by arkadia (Post 1375788)

i build a new nttp gateway from zero using php, C++ and Delphi. it's works very more efficiently than the actual nntp gateway

By the way - don't forget the handshake part between server<->client. The "MODE READER" (that should be placed before the AUTHINFO) is quite important to use, to identify the client as a client. :)

i not understand exactly what you mean? off course i handle the AUTHINFO and all the neccessary nntp protocol !

Quote:

Originally Posted by arkadia (Post 1375807)

i not understand exactly what you mean? off course i handle the AUTHINFO and all the neccessary nntp protocol !

In the current version of the gateway the client "identification" is missing. Since I run a usenet-server that communicates internally with another news-server, the gateway can't handshake correctly without the MODE-string.

A "normal" usenet-client sends "MODE READER" to a server before the authorization, and a usenet-server sends "MODE SERVER" to another server, because of the different article-transfering modes. This usually is based on IP, so if a server is configured to handle feed-traffic, it also treats a local client as a server - which is wrong.

It doesn't happen often, but it happens and may be important for some. :)

ok, i do it ! i send a mode reader just after the connection

wow this mod seem to work fine in 3.6.8 as well as 3.6.7 just needs a few very small query fixes. thanks for this mod.

Would you share the fixes?

This is what I am receiving when I run gateway.php

Code:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html>

<head>

        <title>Domain - Forum Database Error</title>

        <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1" />

        <style type="text/css">

        <!--        

        body { background-color: white; color: black; }

        #container { width: 400px; }

        #message   { width: 400px; color: black; background-color: #FFFFCC; }

        #bodytitle { font: 13pt/15pt verdana, arial, sans-serif; height: 35px; vertical-align: top; }

        .bodytext  { font: 8pt/11pt verdana, arial, sans-serif; }

        a:link     { font: 8pt/11pt verdana, arial, sans-serif; color: red; }

        a:visited  { font: 8pt/11pt verdana, arial, sans-serif; color: #4e4e4e; }

        -->

        </style>

</head>

<body>

<table cellpadding="3" cellspacing="5" id="container">

<tr>

        <td><img src="http://www.mydomain.com/forum/image.php?type=dberror" alt="Database Error" width="48" height="48" /></td>

        <td id="bodytitle" width="100%">Database error</td>

</tr>

<tr>

        <td class="bodytext" colspan="2">The Domain - Forum database has encountered a problem.</td>

</tr>

<tr>

        <td colspan="2"><hr /></td>

</tr>

<tr>

        <td class="bodytext" colspan="2">

                Please try the following:

                <ul>

                        <li>Load the page again by clicking the <a href="#" onclick="window.location = window.location;">Refresh</a> button in your web browser.</li>

                        <li>Open the <a href="/">www.mydomain.com</a> home page, then try to open another page.</li>

                        <li>Click the <a href="javascript:history.back(1)">Back</a> button to try another link.</li>

                </ul>

        </td>

</tr>

<tr>

        <td class="bodytext" colspan="2">The www.mydomain.com forum technical staff have been notified of the error, though you may <a href="mailto:admin@mydomain.com">contact them</a> if the problem persists.</td>

</tr>

<tr>

        <td class="bodytext" colspan="2">&nbsp;<br />We apologise for any inconvenience.</td>

</tr>

</table>



<form><textarea rows="15" cols="70" wrap="off" id="message">Database error in vBulletin 3.6.8:



Invalid SQL:



                        SELECT post.*, thread.*,

                        post.dateline AS postdateline, post.msgid AS postmsgid,

                        thread.title AS threadtitle, post.visible AS postvisible,

                        thread.visible AS threadvisible

                        FROM post as post LEFT JOIN thread as thread

                        ON (thread.threadid = post.threadid

                        AND post.userid = thread.postuserid

                        AND post.postid = thread.firstpostid)

                        WHERE post.isusenetpost = 0

                        AND post.postid &gt; 

                        AND thread.forumid = 27;



MySQL Error  : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'AND thread.forumid = 27' at line 11

Error Number : 1064

Date         : Monday, December 10th 2007 @ 07:00:51 PM

Script       : http://www.mydomain.com/forum/gateway.php?debug=1

Referrer     : 

IP Address   : 000.000.000.000

Username     : admin

Classname    : vB_Database</textarea></form>



</body>

</html>

X vBulletin 3.8.12 by vBS Debug Information
Template Usage: (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (2)bbcode_code_printable (3)bbcode_quote_printable (1)footer (1)gobutton (1)header (1)headinclude (6)option (1)pagenav (1)pagenav_curpage (4)pagenav_pagelink (3)pagenav_pagelinkrel (1)post_thanks_navbar_search (1)printthread (10)printthreadbit (1)spacer_close (1)spacer_open Phrase Groups Available: global postbit showthread	Included Files: ./printthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/class_bbcode_alt.php ./includes/class_bbcode.php ./includes/functions_bigthree.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete printthread_start pagenav_page pagenav_complete bbcode_fetch_tags bbcode_create bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete printthread_post printthread_complete
Messages:

Phrase Groups Available:

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.01841 seconds Memory Usage 1,764KB Queries Executed 10 (?)
More Information
Template Usage: (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (2)bbcode_code_printable (3)bbcode_quote_printable (1)footer (1)gobutton (1)header (1)headinclude (6)option (1)pagenav (1)pagenav_curpage (4)pagenav_pagelink (3)pagenav_pagelinkrel (1)post_thanks_navbar_search (1)printthread (10)printthreadbit (1)spacer_close (1)spacer_open Phrase Groups Available: global postbit showthread	Included Files: ./printthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/class_bbcode_alt.php ./includes/class_bbcode.php ./includes/functions_bigthree.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete printthread_start pagenav_page pagenav_complete bbcode_fetch_tags bbcode_create bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete printthread_post printthread_complete
Messages: