Word Processors & vBulletin UTF-8 Stripped Posts (Possible Fix?)

[orth]

Hi all, I'm fairly new to the inner workings of vBulletin and I found myself tackling a bug when users would post directly from MS Word or Appleworks. Now these programs have the SmartQuotes and AutoCorrect options and those seemed to be Windows 1252 encodings so when the user would enter their text with copy and paste, what was happening was the moment the post hit a non UTF-8 character, the rest of the post would get cut off. I didn't have much luck finding any help with some vBulletin googling. After a lot of searching around though I found a helpful post on the php forums. All it required I do was add the following bolded code in includes/class_core.php:

function htmlspecialchars_uni($text, $entities = true)
{
        $text = mb_convert_encoding($text,"HTML-ENTITIES","auto");
        return str_replace(
                // replace special html characters
                array('<', '>', '"'),
                array('&lt;', '&gt;', '&quot;'),
                preg_replace(
                        // translates all non-unicode entities
                        '/&(?!' . ($entities ? '#[0-9]+' : '(#[0-9]+|[a-z]+)') . ';)/si',
                        '&amp;',
                        $text
                )
        );
}

As I said before I'm not too familiar with the vBulletin innerworkings nor PHP and I just want to make sure I'm not invalidating any security measures with HTML using this? If I'm not then maybe someone else who is having the same problems might find this useful.

Thanks for any insight,
orth

[Asilouhuette]

I have this problem as well. It seems to work fine for previews, but once you actually make the post it screws everything up.

Is that how you experienced it as well?

I havn't seen any other recommendations on this, so unless someone (From the vB Team maybe?) advises otherwise, I think I'll use this fix. Anyone know if this will be an included fix in future versions?

Thanks

[Dismounted]

You should report bugs to the vBulletin Bug Tracker (http://www.vbulletin.com/forum/project.php?projectid=6).

[Asilouhuette]

I tried this fix and unfortunately it didn't work for me.

I'm not sure that it is really a bug so much as a poor choice of character encoding on my part. Choosing UTF-8 may have let me avoid this.

I tried fixing it myself, with varying amounts of success. I tried placing an str_replace() that replaces smartquotes and other special characters with their $blah; counter parts. This gave me some improvement, however, I couldn't find the most optimal place to put this. I tried having it change the $dataman->['pagetext'] after the various _presave hooks but that didn't fix all cases. It would only work with edits || new posts, not both, for example.

Could anyone point me to which hook would be caught by all posts/post edits and which variable would be best to change at that hook?

I think it would be fairly trivial to do this if I didnt use a hook, but that of course makes maintaing updates much harder. Here is the str_replace I'm using...

PHP Code:

    $wordentities = array(chr(128), chr(130), chr(131), chr(132), chr(133), chr(134), chr(135), chr(136), chr(137), chr(138), chr(139), chr(140), chr(145), chr(146), chr(147), chr(148), chr(149), chr(150), chr(151), chr(152), chr(153), chr(154), chr(155), chr(156), chr(159));                
    $htmlwordentities = array('€', '‚', 'ƒ', '„', '…', '†', '‡', 'ˆ', '‰', 'Š', '‹', 'Œ', '‘', '’', '“', '”', '•', '-', '—', '˜', '™', 'š', '›', 'œ', 'Ÿ');
    return str_replace($wordentities, $htmlwordentities, $text); 


Thanks in advance.

[dnetzer]

Just stumbled upon this post. Thanks for the insightful information people. Have you since, found a better solution? I see people inserting text which is cut & pasted from Word docs. What a mess.