Strange UNICODE issues with vB 3.6.8

[evannn]

Hi,

We've started a non-English sub forum.

I've changed the encoding scheme to UTF-8 in Language and Phrase. Generally the non-English text shows up fine.

I noticed some weird issues:

For instance, in a post, certain unicode coded characters refuses to show up. It will appear only as a [] - like character. We are sure that this is a very common character in the chinese language.

Secondly, sometimes the first unicode character in the post appears as a box like structure.

Am I missing anything here?

Pleae advise.

Thanks.

[Marco van Herwaarden]

What is your database character set?

[evannn]

latin_swedish_ci.

I've tried to change all the collations to UTF-8 in a testing system using a database dump from live server.

It corrupts the database totally. Close to 5GB of data.

Are there any other ways?

Please advise.

[Marco van Herwaarden]

Please PM me your email address and i will sent you a small script that might help.

[evannn]

Thanks Marco. It seems to work well.

I've tried it on my testing machine and it seems to work well. Will do another round of testing on the live server using a duplicate db.

I noticed how vB stores non english data - it converts everything to HTML entities before it saves into the database.

I meant, are there any ways which the data will be stored into the database the way it is typed and written? Instead of vB converting it to #6734 ?

[Marco van Herwaarden]

It is not vB that does the converting.

PS Good to hear the script was usefull.