Understanding post content unicode

[av8or1]

Hi-

I am trying to include foreign language snippets into post content during post creation via script execution. I have read the information about the foreign language downloads that you can install in your forum and have read a few posts regarding unicode processing in vB, but I haven't found a way of doing this yet. Thus this post requesting guidance.

I haven't altered the unicode settings for my forum, mostly because my testing has shown that this didn't have an effect on the post content, though I could have missed something along the way. This is a summary of what I tested:

To begin I simply added a new post interactively via the standard editor on my test forum. I cut and pasted the text, which is cyrillic BTW, directly into the message and pressed the submit button. The text rendered correctly:

катушки

and upon viewing the source I saw that the standard HTML entities for the unicode characters that correspond to the entered text were displayed:

&#x43A&#x430&#x442&#x443&#x448&#x43A&#x438

So the question I have is how to replicate this via a script? The data that my script receives has these characters represented in their unicode \u0XXX form (at least I think that's the standard form, though I seem to recall the %uXXX form too) which is easily convertable to the HTML equivalent. However when my script does that and then submits the post (via a post DM object) all I see are the above HTML entities in the textual content of that post. And of course the same result occurred when leaving them as \u0XXX.

So I dug around in the code. I tried applying html_entity_decode to the body of the post prior to submitting it, but that didn't have any effect. I dug further. I found a couple of interesting items that I was going to attempt next, specifically:

unhtmlspecialchars (vB function)
htmlspecialchars_decode (php function)

The unthmlspecialchars would need the second parm set to true or else it won't decode unicode entities and I only saw this done in a couple of places within all of the vB code.

Anyway it was about midnight when I found these and so I haven't tried them yet, mostly because I'm not sure they will work or if it is even the correct approach to the problem.

So in the end I'd just like to ask: has anyone else dealt with this issue already and if so, can you describe how you solved it? I conducted a few searches on vb.org/forum but didn't find anything.

Thanks!

Jerry