vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vBulletin 3.0 Full Releases (https://vborg.vbsupport.ru/forumdisplay.php?f=33)
-   -   NNTP Gateway for Usenet ( Newsgroups ), Mailing Lists (https://vborg.vbsupport.ru/showthread.php?t=65152)

AWS 06-27-2004 02:44 AM

Quote:

Originally Posted by Imhotep
:ermm: My forums stopped grabbing new Usenet messages on 6/21. I'm trying to finger out exactly where the log jam has occured. My cron job is still going off like clockwork.

This is the entire content of my cron results minus the header garbage.

Code:

Gateway version 2.1.0  4 group(s) gatewayed.

Connecting to server, server says: 200 News.GigaNews.Com

Server responded after user name: 381 more authentication required

Server responded after password: 281 News.GigaNews.Com

Info for comp.graphics.apps.lightwave at news-40.giganews.com:
211 12589 237200 249788 comp.graphics.apps.lightwave

Getting message number 249690:


Any ideas as to what is happening? It looks like it only hits one group... then stops. ? :ermm:

Quote:

Getting message number 249690:
The message is not there and the script thinks it has reached the end. Go to newsgroup settings and change the id number and run the script again.

Imhotep 06-27-2004 05:16 AM

awesome! thank you very much

kmike 06-27-2004 07:13 AM

Quote:

Originally Posted by AWS
Quote:

Getting message number 249690:
The message is not there and the script thinks it has reached the end. Go to newsgroup settings and change the id number and run the script again.

While you posted good workaround for this, the root of this problem is actually completely different. We encountered the same problem at about the same time, and the real cause was PHP crashing on the huge spam messages crossposted to several newsgroups. Here's an example (5000+ lines!):

http://groups.google.com/groups?dq=&...ing.google.com

The quick fix is to limit amount of lines we get from the news server per message.
After the line 128 of includes/nntp.php, which reads:
PHP Code:

        $post ''

add this line:
PHP Code:

        $lc 0

Then find this code fragment starting on line 134:
PHP Code:

            if ($line == ".") {
                break;
            } else {
                
$post .= $line ."\n";
            } 

Change it to:
PHP Code:

            if ($line == ".") {
                break;
            } else if (
$lc++ < 300) {
                
$post .= $line ."\n";
            } 

Change 300 here to the amount of lines from news posting you want to allow to get into forum's message.

Enjoy :)

Princeton 06-27-2004 03:25 PM

A word (title or message) that long will break a fixed width forum. It will even make a liquid layout appear "bad".

The fetch_word_wrapped_string function does not work on long words with periods, commas, special characters. This is a bug that has been reported.

Until there's a fix ... you should use the fetch_trimmed_title on titles (showthread/forumdisplay). fetch_trimmed_title function works with the $vboptions[wordwrap] settings.

If you do not want to use the fetch_trimmed_title function...
I use this on my cms: (mainly for right/left column):
Code:

function gtp_strip_title($str, $start, $end) {
        if (strlen($str) > $start){
              $str = substr($str, 0, $end)."...";
        }
      return $str;
 }

I suggest having an option to remove (not insert in db) messages with long titles.
All messages with long words (titles) are 100% spam.

kkiely 06-28-2004 02:25 AM

I am a newbie. With this script I keep hearing people saying to "run a cron job"

1. What is "cron"? I assume it is some type of scheduling task system.
2. What will it do with this script? does it auto-import the posts?

Odoin 06-28-2004 02:38 AM

A cron is where you go downtown and play the shell game and cron people out of their money...

Just kidding... many providers that offer a Control Panel have some way to setup a cron job. You are right what a cron is... It runs a script or program and a set time.

In this case you want it to run the gateway.php script to pick up and send new NNTP messages. You could do this your self by pointing your browser to the gateway.php file... however if you forget or are gone for a few days the new posts will not be updated. Another factor is on large NNTP groups it takes a lot more time if ran once a day rather then every hour.

chopperp1 06-29-2004 08:22 PM

Hello to All
I'm trying to get this hack working on company's web server. Everything was pretty much straight forward till I got to the point of running gateway.php. That's what I get:
"...pop3 gateway

Gateway version 2.1.0 1 group(s) gatewayed. Bad User Name!Retrieving Protocol E-mails from the 'dl360mail3.yamner.com' pop3 server ..."

I tripple checked all settings and everything looks good. User account for pop3 is correct including password. Web server talks to mail via local subnet (no filtering there). Can't figure out why this thing can't log in.
any clues? :ermm:

lierduh 06-29-2004 09:54 PM

Quote:

Originally Posted by chopperp1
Hello to All
I'm trying to get this hack working on company's web server. Everything was pretty much straight forward till I got to the point of running gateway.php. That's what I get:
"...pop3 gateway

Gateway version 2.1.0 1 group(s) gatewayed. Bad User Name!Retrieving Protocol E-mails from the 'dl360mail3.yamner.com' pop3 server ..."

I tripple checked all settings and everything looks good. User account for pop3 is correct including password. Web server talks to mail via local subnet (no filtering there). Can't figure out why this thing can't log in.
any clues? :ermm:

The best way is to log into your web server and do a telnet from there:

#telent dl360mail3.yamner.com 110
#USER your-username
#PASS your-password
#STAT

I suspect your pop3 server does not like the pop3 access from a local subnet. You gave a public domain name, yet accessing through a local private net. Either routing or pop3 server's "allowed access" can cause your problem.

john_rsd 06-29-2004 10:05 PM

Ive been testing this for a few days now.

I have found that the script has crashed twice causing the following error

Quote:

Database error in vBulletin 3.0.1:

Invalid SQL: INSERT INTO attachment SET dateline = '1088526838',
filename = '',
filedata = 'GIF89a?\0,\0?\0

<<snipped here>>
thumbnail = '',
filesize = + 0,
postid = 202
mysql error: You have an error in your SQL syntax. Check the manual that corresponds to your MySQL server version for the right syntax to use near '+ 0,
postid = 202' at line 5

mysql error number: 1064

Date: Tuesday 29th of June 2004 09:33:59 AM
Script: http://proteluser.com/bbs3/gateway.php
Referer:
Username: Site_Admin
But the script seems to have went so far then went pear shaped, at this point threading by title was completely non-effective and the 22 messages were not threaded at all despite having subject headers matching exactly. See screenshot attached.

There also seems an issue with handling possible EOL characters as i got a message like this inserted to the forums. Notice the '=' inserted at the end of a line before a return.

Quote:

I've just been able to use DXP for a complete job and feel that I can now c=
omment on a few things that I've found.

Of course some of this may have a new 'DXP' way of doing things that I don'=
t know of yet.

In 99Se I could 'RMB' cut, copy or paste when editing many parameters such =
as drawing number in the schematic editor, I can't do this anymore and inst=
ead must use the more cumbersome ctrl C / ctrl V method.

Usually when editing text I want to change the text itself - in 99SE, doubl=
e clicking on a text string opened an editor panel with the 'text' selected=
.. This was very good for text editing as all I needed to do was to double c=
lick, type in the new value and then enter. Now I double click and then hav=
e to find and select the 'value' to change.

It should have looked like this, at least this is how outlook displays it in the mail box.

Quote:

I've just been able to use DXP for a complete job and feel that I can now comment on a few things that I've found.

Of course some of this may have a new 'DXP' way of doing things that I don't know of yet.

In 99Se I could 'RMB' cut, copy or paste when editing many parameters such as drawing number in the schematic editor, I can't do this anymore and instead must use the more cumbersome ctrl C / ctrl V method.

Usually when editing text I want to change the text itself - in 99SE, double clicking on a text string opened an editor panel with the 'text' selected. This was very good for text editing as all I needed to do was to double click, type in the new value and then enter. Now I double click and then have to find and select the 'value' to change.
This message was sent as plain text.

This was the second message inserted in that batch. The first message is uneffected and was threaded correctly.

Is there any compatability issues with mysql or php versions ???

Any help appreciated

john_rsd 06-29-2004 11:57 PM

After looking at the messages retreived from the pop3 mailbox on a different account to the one used for gatweway.php i noticed that there was a few messages originally in HTML and with gif as attachments.

Could the above error be an issue with trying to insert the attachment?

Attachments are usually frowned upon on mailing lists which is why I failed to notice it when making my last post.

Is there a quick fix to simply drop attachments completely?

lierduh 06-30-2004 12:02 AM

Quote:

Originally Posted by john_rsd
After looking at the messages retreived from the pop3 mailbox on a different account to the one used for gatweway.php i noticed that there was a few messages originally in HTML and with gif as attachments.

Could the above error be an issue with trying to insert the attachment?

Attachments are usually frowned upon on mailing lists which is why I failed to notice it when making my last post.

Is there a quick fix to simply drop attachments completely?

Are you using the original gateway.php or you have installed Kevin's mini-hacks? Not easy to find the problem if we are not taling about the same script. Kevin has re-written the attachment routine which I will include with next version.

alesis404 06-30-2004 05:52 AM

Hello, in the read me you mention to change this in the "nav" template if you would like usenet posts from not showing up in new postings

Code:

Find:
<td class="vbmenu_control"><a
href="search.php?$session[sessionurl]do=getnew">$vbphrase[new_posts]</a></td>

Change it to:

<td class="vbmenu_control"><a
href="search.php?$session[sessionurl]do=getnew&exclude=22,23,24,25,26,27">$vbphrase[new_posts]</a></td>

I have searched my *nav* template and the closet match looks like this

Code:

<td class="vbmenu_control"><a href="search.php?$session[sessionurl]do=getnew" accesskey="2">$vbphrase[new_posts_nav]</a></td>
*Perhaps my templates are newer than when this was released? Can someone please confirm if the code for editing "nav" template has indeed changed ,thanks.

john_rsd 06-30-2004 07:12 AM

Quote:

Originally Posted by lierduh
Are you using the original gateway.php or you have installed Kevin's mini-hacks? Not easy to find the problem if we are not taling about the same script. Kevin has re-written the attachment routine which I will include with next version.

Sorry for missing information .

I used the original gateway.php from the first post.

I added Kevins mod for thread by subject. The forum is a test one so does not have users accessing at same time as script runs. Script was run directly from browser, the forums only content is the messages imported from the mailing list. Max 20 messages per day, max 10-15 messges per batch.

lierduh 06-30-2004 10:34 AM

Quote:

Originally Posted by alesis404
Hello, in the read me you mention to change this in the "nav" template if you would like usenet posts from not showing up in new postings



Code:

<td class="vbmenu_control"><a href="search.php?$session[sessionurl]do=getnew" accesskey="2">$vbphrase[new_posts_nav]</a></td>
*Perhaps my templates are newer than when this was released? Can someone please confirm if the code for editing "nav" template has indeed changed ,thanks.

This is the right one. It used to be "new_posts", the latest vB3 has changed to "new_posts_nav".

alesis404 06-30-2004 10:40 AM

Thank's lierduh!

chopperp1 06-30-2004 11:55 AM

Thanks for quick reply.
I already tried telnet from webserver- no problems. I used FQDN because we host our internal dns servers, anyway I get same error when I specify by IP.
__________________________________________________ _________________
+OK Microsoft Exchange 2000 POP3 server version 6.0.6249.0 (dl360mail3.yamner.co
m) ready.
+OK
+OK User successfully logged on.
+OK 21 32830
__________________________________________________ __________________

Is there anything else I should look for?
Thx

lierduh 06-30-2004 12:30 PM

Quote:

Originally Posted by john_rsd
Sorry for missing information .

I used the original gateway.php from the first post.

I added Kevins mod for thread by subject. The forum is a test one so does not have users accessing at same time as script runs. Script was run directly from browser, the forums only content is the messages imported from the mailing list. Max 20 messages per day, max 10-15 messges per batch.

I have merged KevinM's attachment routine into the gateway.php. Please try out.

You now need to run

http://forums.yourdomain.com/gateway.php?debug=1

Without the debug=1, the script won't output anything on the screen.

lierduh 06-30-2004 12:37 PM

Quote:

Originally Posted by chopperp1
Thanks for quick reply.
I already tried telnet from webserver- no problems. I used FQDN because we host our internal dns servers, anyway I get same error when I specify by IP.
__________________________________________________ _________________
+OK Microsoft Exchange 2000 POP3 server version 6.0.6249.0 (dl360mail3.yamner.co
m) ready.
+OK
+OK User successfully logged on.
+OK 21 32830
__________________________________________________ __________________

Is there anything else I should look for?
Thx

Please include the exact letters you typed during the telnet session and the response the server returned.

chopperp1 06-30-2004 12:50 PM

Quote:

Originally Posted by lierduh
Please include the exact letters you typed during the telnet session and the response the server returned.

+OK Microsoft Exchange 2000 POP3 server version 6.0.6249.0 (dl360mail3.yamner.co
m) ready.
+OK
USER <domain>\<user>
PASS <password>
+OK User successfully logged on.
+OK 21 32830

I also tried forward slash or user@domain.com - didn't work

lierduh 06-30-2004 01:04 PM

Quote:

Originally Posted by chopperp1
+OK Microsoft Exchange 2000 POP3 server version 6.0.6249.0 (dl360mail3.yamner.co
m) ready.
+OK
USER <domain>\<user>
PASS <password>
+OK User successfully logged on.
+OK 21 32830

I also tried forward slash or user@domain.com - didn't work

How long is <domain>\<user> ?

The user name field is limited to 32 characters. If you need longer, you can modify the database to increase the char length.

If you can access the DB directly, eg. using phpmyadmin, check what exactly is stored in the DB for the user name.

Looks like it is either the length or the "\" causing the issue.

chopperp1 06-30-2004 03:13 PM

Quote:

Originally Posted by lierduh
How long is <domain>\<user> ?

The user name field is limited to 32 characters. If you need longer, you can modify the database to increase the char length.

If you can access the DB directly, eg. using phpmyadmin, check what exactly is stored in the DB for the user name.

Looks like it is either the length or the "\" causing the issue.


domain as you noticed is "yamner' and user name is 5 characters. I just checked "nntp_groups" table in mysql and I don't see any issues there (server lenght=64; username=32; password=32) all values are correct . :ermm:
But I noticed one thing, when I go to admincp>Execute SQL Query I get the message "You are not authorized to execute SQL queries"- I'm logged in as a superadmin. Do you think this might be my problem? If yes, what file should I modify?
Thx

john_rsd 06-30-2004 03:26 PM

Quote:

Originally Posted by lierduh
I have merged KevinM's attachment routine into the gateway.php. Please try out.

You now need to run

http://forums.yourdomain.com/gateway.php?debug=1

Without the debug=1, the script won't output anything on the screen.

Cool :)

Ill try it, is this just the original php file + Kevins mod or
an updated gateway.php + Kevins mod?

Ill run it later and see if I can recreate it.

KevinM 06-30-2004 05:11 PM

Quote:

Originally Posted by john_rsd
Cool :)

Ill try it, is this just the original php file + Kevins mod or
an updated gateway.php + Kevins mod?

I updated how part of the script that handles the attachments works (this is not related to any previous mini-hacks I have released). There are 3 minor improvements:
1. The script will now handle attachments as 'save as files' if that is how your version of vb is set-up.
2. It checks attachment permissions against the forum (allow attachments or not) and extension types (as set up in control panel).
3. On some flavours of php installations the making of thumbnails was problematic. It now follows vb's standard way of thumbnail creation.

There may be other improvements made by Lierduh ;)

vprp 06-30-2004 07:08 PM

Quote:

Add a smallint field on the database record such as "is_usenet" and set it to '1' for the forums that are.
How do you add a smallint field?

alesis404 06-30-2004 08:45 PM

Can someone please tell me how to properly stop attachments being made with posts, I get constant errors sent to my email everytime gateway.php run's , I tried to comment out a small piece in gateway.php and this stopped the attachments and the posts so I must have did something wrong. I re-edited and am getting attachment errors again.

john_rsd 06-30-2004 09:18 PM

Quote:

Originally Posted by KevinM
I updated how part of the script that handles the attachments works (this is not related to any previous mini-hacks I have released). There are 3 minor improvements:
1. The script will now handle attachments as 'save as files' if that is how your version of vb is set-up.
2. It checks attachment permissions against the forum (allow attachments or not) and extension types (as set up in control panel).
3. On some flavours of php installations the making of thumbnails was problematic. It now follows vb's standard way of thumbnail creation.

There may be other improvements made by Lierduh ;)

Kevin

So is the mini hack for thread by subject enhancement is now integrated or will it need updated again for this version of gateway.php?

john_rsd 06-30-2004 10:06 PM

Its Ok Kevin, I answered my own question (just tried it)

If i add the thread by subject mini hack I get

Parse error: parse error in /home/protelus/public_html/bbs3/gateway.php on line 401

Without it its fine.

KevinM 07-01-2004 07:42 PM

Quote:

Originally Posted by john_rsd
Its Ok Kevin, I answered my own question (just tried it)

If i add the thread by subject mini hack I get

Parse error: parse error in /home/protelus/public_html/bbs3/gateway.php on line 401

Without it its fine.

I will update it when Lierduh makes an 'official' release.

lierduh 07-01-2004 10:17 PM

Quote:

Originally Posted by chopperp1
domain as you noticed is "yamner' and user name is 5 characters. I just checked "nntp_groups" table in mysql and I don't see any issues there (server lenght=64; username=32; password=32) all values are correct . :ermm:
But I noticed one thing, when I go to admincp>Execute SQL Query I get the message "You are not authorized to execute SQL queries"- I'm logged in as a superadmin. Do you think this might be my problem? If yes, what file should I modify?
Thx

With pop.php file, please find:

$this->_write("USER $this->username");

and insert the following line before that:

echo "=========USER Name: " . $this->username . " ==========";

This will show the user name entered by the script. I want to confirm if the right username is used by the script at this stage.

Also with gateway.php, find:

$pop_connect = $pop3->connect();

and add the following line after this:

logging($pop3->showlog());

This will show the entire pop3 connection log. No fix at this stage, just need to find out the reason behind this first.

lierduh 07-01-2004 10:19 PM

Quote:

Originally Posted by alesis404
Can someone please tell me how to properly stop attachments being made with posts, I get constant errors sent to my email everytime gateway.php run's , I tried to comment out a small piece in gateway.php and this stopped the attachments and the posts so I must have did something wrong. I re-edited and am getting attachment errors again.

Have you tried the script I posted in last page?

john_rsd 07-02-2004 12:17 AM

Quote:

Originally Posted by KevinM
I will update it when Lierduh makes an 'official' release.

Cool :cool:

Thanks

alesis404 07-02-2004 05:54 AM

Quote:

Originally Posted by lierduh
Have you tried the script I posted in last page?

Lierduh are you referring to post #297 of this thread? If so do i just upload the new gateway.php and this will stop the attachments or will stop the errors?

john_rsd 07-04-2004 12:17 AM

lierduh

I am still having trouble with the new gateway.php script with file attachments.

How aware is the script about which type of attachment it is?

The forum has no file upload permissions. So I tested this by emailing the account which is used for gateway with a big file and it crashed (I suspect timeouts)

So i send it a very small file to test this theory, it was a small zip file only 36kB and instead of dropping the attachment as it should, it inserted it as text below the final characters of the message, see screenshot atatched.

Anything you need me to do for you to trap what is going on?

chopperp1 07-06-2004 02:42 PM

Quote:

Originally Posted by lierduh
With pop.php file, please find:

$this->_write("USER $this->username");

and insert the following line before that:

echo "=========USER Name: " . $this->username . " ==========";

This will show the user name entered by the script. I want to confirm if the right username is used by the script at this stage.

Also with gateway.php, find:

$pop_connect = $pop3->connect();

and add the following line after this:

logging($pop3->showlog());

This will show the entire pop3 connection log. No fix at this stage, just need to find out the reason behind this first.

Sorry for late response, I was away for few days.
Here's what I got after adding those lines
"...pop3 gateway

Gateway version 2.1.0 1 group(s) gatewayed. =========USER Name: yamner\******==========+OK Microsoft Exchange 2000 POP3 server version 6.0.6249.0 (dl360mail3.yamner.com) ready. USER yamner\****** -ERR Protocol error. Bad User Name!Retrieving Protocol E-mails from the '209.178.209.17' pop3 server ..."

I had to hide a user name, but real log shows correct one. Also, as you can see, this time I used external IP for pop server - different isp than board is hosting on...

trackpads 07-06-2004 06:00 PM

What is the maximum characters that we can put in the Killfile exclude field? I think I ran out already, is there a way to expand it?

Thanks again!

-Jason

kall 07-07-2004 07:40 AM

Whoooops.

When upgrading to 3.0.2, don't forget that the script will repost all of the threads already in your selected forum.

*starts composing apology emails*

KevinM 07-07-2004 09:44 AM

Quote:

Originally Posted by trackpads
What is the maximum characters that we can put in the Killfile exclude field? I think I ran out already, is there a way to expand it?

It's limited to 128 characters (which is what the original scripts default settings are), and, like yourself I found it too small.

Run the following sql query and you can have up to 64000 characters (give or take!)

PHP Code:

ALTER TABLE `nntp_settingsCHANGE `value` `valueMEDIUMTEXT DEFAULT NULL  

It's possible that in future upgrades of gateway.php the script may revert back to the original settings (as it auto-installs), so if there are any future gateway updates that you install you will have to run this query again after installing (and keep a copy of what's in the killfile beforehand as this may be erased).

kall 07-07-2004 10:12 AM

And here's what one nice person told me:

Quote:

News has a well-defined protocol (NNTP - RFC977 etc) and if you have a program which retrieves and/or send articles then it should retain the very important "Message-ID" header which each article contains. Then if it accidentally tried to post an old article it would get rejected because of the conflicting Message-ID (unique for each article). Otherwise it would generate a unique Message-ID for a new article and present it with the "IHAVE" nntp command.
*sighs*

AWS 07-07-2004 10:18 PM

Quote:

Originally Posted by kall
Whoooops.

When upgrading to 3.0.2, don't forget that the script will repost all of the threads already in your selected forum.

*starts composing apology emails*

Are you saying when you upgraded to 3.0.2 the gateway script sent all your posts to usenet from the usenet forums?

How can this happen?
You have to run gateway.php to upload new posts and it has the message id already set so duplicates shouldn't be uploaded. The only way I can see this happening is if the upgrade to 3.0.2 made all posts look as new posts or somehow the upgrade lost all id numbers in the db.

kall 07-07-2004 10:22 PM

Quote:

Originally Posted by AWS
The only way I can see this happening is if the upgrade to 3.0.2 made all posts look as new posts or somehow the upgrade lost all id numbers in the db.

*nods*

That's what happened, because I ended up doing the upgrade by installing afresh and importing all of the posts etc using Impex..obviously losing the columns that denoted the threads as being 'old'.


All times are GMT. The time now is 02:25 AM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01870 seconds
  • Memory Usage 1,914KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (5)bbcode_code_printable
  • (5)bbcode_php_printable
  • (30)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (4)pagenav_pagelink
  • (1)pagenav_pagelinkrel
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (40)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 

Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • pagenav_page
  • pagenav_complete
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete