vCharset Converter 1.0.0 BETA 2

[Milad]

THE 3.5 BRANCHE IS CLOSED, A NEW VERSION FOR 3.6 HAD BEEN RELEASED.

To use this modification, please update to 3.6.x and use this.

By syrianmeds.net (http://www.syrianmeds.net/)

Description: This script converts the database of vBulletin forums (from and to) any of the charsets that are supported by iconv. also it has the ability to convert the tables of the hacks you had installed them.

Caution: This is BETA PRODUCT, provided "As Is" without warranties, Test it using a copy of your vBulletin database not using your live vBulletin, Since this is a BETA PRODUCT use it for testing only.

Why this product?
My forums encoding is windows-1256, so I can't get full advantages of AJAX functions like the quick reply and the quick edit without some encoding problems.
I decided to convert my encoding and build this script. and I'll convert to utf-8 (Done).
This product was tested on a copy of real live vBulletin and it seems to be good.

Requirements:
vBulletin 3.5.4+
iconv php extension is installed on your system, or your PHP version is PHP5


Supported Encodings:
See http://www.gnu.org/software/libiconv/#introduction

Installation:

Upload files to their respective locations.
Import the file product product-charset_converter.xml (Choose Yes when Upgrade)
Refresh your admincp, go to vCharset Converter->Settings and set (the Source Charset Encoding & the Destination Charset Encoding).
Go to vBulletin Options -> Censorship Options -> Blank Character Stripper, empty its value and click save.


How to use?

Close your forum.
Backup your database.
Add a new language and call it English for example, its charset is your Destination Charset Encoding.
Set your options (profile.php?do=editoptions) to use it as a default language.
Change the charset of your old langauge into Destination Charset Encoding.
Go to vCharset Converter -> Manage tables. I'll leave this to you to explore it. But edit the userfield table and the custom fields to it.
Go to vCharset Converter -> vCharset Converter. if you see a message like: The iconv PHP module is not presented, please contact your system administrator to enable/install it. this means you need to install iconv.
You will see a drop-down menu. select the table you want to convert, set the number of items to convert per cycle (default 1000) and press submit.
You have to do this with each table in the drop-down menu.
Test your forum search for Non-English letters, if you don't get the expected results, read the "Search Problems".
Open your forum.
After finishing tell me what are your impression, idea and faced problems.

Search Problems
What is the problem? After converting you may get no (well-expected) results when doing search, I had faced this problem after converting to UTF-8.
What is my circumstances? Mysql 4.1.x, Full-Text search and I converted to utf-8.
What is the reason? Full-Text search has Restrictions (http://dev.mysql.com/doc/refman/4.1/en/fulltext-restrictions.html) that say: "all columns in a FULLTEXT index must use the same character set and collation.".
So if you use fulltext search with MySQL 4.1.x and convert to utf-8 perform these queries: (after converting post and thread tables)

ALTER TABLE post MODIFY pagetext mediumtext COLLATE utf8_unicode_ci
ALTER TABLE post MODIFY title VARCHAR(250) COLLATE utf8_unicode_ci
ALTER TABLE thread MODIFY title VARCHAR(250) COLLATE utf8_unicode_ci

Dont forget to add your table prefix if you have

- If you use vBulletin built-in search engine (with any MySQL version), then consider to Rebuild Search Index (Maintenance -> Update Counters -> Rebuild Search Index)
You may read this thread http://www.vbulletin.com/forum/showthread.php?t=191315

Email Problems
After the conversion, my users had reported that they couldn't read the notification emails. the emails are sent with utf-8 encoding.
Not all the users do know how to change the encoding of a page. you may learn them, this is the only solution.
Problem had been faced with Hotmail (confirmed by me) and Yahoo (unconfirmed).
No problem with Gmail (confirmed by me).

Known issues:

Users titles had gone away after converting. (Still exists in beta 2)


Versions

1.0.0 BETA 1 (initial release). (11. Jun 2006)
1.0.0 BETA 2 new features: (4. Aug 2006)

Manage Tables: Now you can add or edit your tables, this will make converting of additional tables very easy.
New built-in images to mark the tables:

50952 Table is ready to be converted.
50955 Table is uknown.
50954 No need to convert this table.
50953 Table Converted :)

Now you can see all the tables in your database, with their statues and you have the ability to change this status to "no need" or you can add the fields and the primaryid field, or you can edit the information of any table before you go on converting it.



Forums had been Converted using this tool: (PM me to list your forum)

http://www.basharialep.com/forum/ from windows-1256 into UTF-8


Support is given for who clicked "install" only.

Click https://vborg.vbsupport.ru/ (https://vborg.vbsupport.ru/vborg_miscactions.php?do=installhack&threadid=118308) to recieve updates.

Nominate it for Mod o.t. Month (https://vborg.vbsupport.ru/vborg_miscactions.php?do=nominatehotm&t=118308) if you love it.

[Milad]

Post reserved.

[Barakat]

thanks Milad i will test it soon

[Panagioths]

Your script is extremely good. During the last 4 days I am trying to convert the database of my forum from greek ISO to utf8, I tried hundrend ways without the desired output and when I installed the vCharset all conversions made automagically and without any problems as fas as I can see now.

I have a question: I removed a part of data of my database by accident and now in product list (in admincp) vCharset is not shown anymore (in order to uninstall it since the convert is done). Also menu "vCharset" in admincp is not shown anymore. But I've got a table "charset_convert" in my database so I assume there are some parts of vCharset installtion still in forum.

Is there a sql command that I can run in order to remove any part of vCharset in my forum?

[Milad]

Thanks for your feedback

I assume you ran the script on a test board not on live one.

vCharset converter must remain in your database after finishing, it doesn't uninstal itself, you can uninstall it manually. yes you can uninstall the product, drop the table charset_convert and delete the product files.

Because you delete something in your database, you don't see the product, I think you deleted or epmtied product table.

I advise you to return to your backup, and test my script many times before you make decision to go ahead with it.

Remember I am still considering this product as a BETA product.

Thanks for your feedback again. :)

[Milad]

Also if you have some hacks, extensions or plugins that collect some text information in the database, you have to convert them too, this will be explained after the gold is released.

[Omranic]

thanks alot Milad
thats extremely good

Go on developing this
we need to see a strong stable version as soon as posible


;)

[Milad]

Thanks

Please to keep yourselves updated press install (https://vborg.vbsupport.ru/vborg_miscactions.php?do=installhack&threadid=118308).

[blacklude]

Thanks for your feedback

I assume you ran the script on a test board not on live one.

vCharset converter must remain in your database after finishing, it doesn't uninstal itself, you can uninstall it manually. yes you can uninstall the product, drop the table charset_convert and delete the product files.

Because you delete something in your database, you don't see the product, I think you deleted or epmtied product table.

I advise you to return to your backup, and test my script many times before you make decision to go ahead with it.

Remember I am still considering this product as a BETA product.

Thanks for your feedback again. :)

is there any reason that the convertor must stay after finishing the convertion?
thanks!

[Milad]

is there any reason that the convertor must stay after finishing the convertion?

Never

But i want you to know that the product doesn't uninstall itself after the converting, you should uninstall it yourself.

Regards

[Milad]

Any ideas? suggestions? or bugs?

[e7lew]

ميلاد جربته لكن طلع خطا في اكثر من جدول والتجربة كانت على 3.6.0
وبعدين لما اعرض السورس من المصدر تطلعلي جمل غريبة
ط?ظ?ظ?ظ?ط?ط?ظٹط?: 1, ط?ظ?ظ?ط?ط?ط?ظ?ط?طھ: 5, ط?ظ?ط?ط?ط?ط?ط،: 1<
وبعدين الرد السريع ما يرسل اضغط عليه يرجع مكانه لما اسوي رفرش يطلع الرد بالموضوع
وهذه صورة بالمرفق للجداول الي ما صارت او ما تحولت
اعتقد يمكن من اصدارة النسخة البيتا الجديدة
عندي منتدى ضخم بجرب عليه اصداره 3.4.5
وبردلك خبر

thanks :) :banana:

[Milad]

This is for vB3.5.x only, for the source code it's normal to see it like ': 5, ط?ظ„ط?ط?ط?ط'

also do you change the charset html for the languages into utf-8? I think you didn't.

Please don't use it on live boards, and only for 3.5.x

Thanks for your feedback, and I'm waiting for more :)

[Almotmaiz.Net]

Great work mr. Milad i'll test it in forums ..

be right back .. ..

[Lizard King]

I hope you'll release the gold version soon.

[neuston]

I had a test, here is my report:

my site is designed for Chinese-language user. Unfortunately, there are two sets of chinese data with different charsets, utf8 and Big5. I wanted to change all the data to utf8, followed your steps. However, your convertor not only cannot convert the data with charset big5 to utf8, and also change the data with charset utf8 to monster words.

I don't know, I might take some wrong steps :confused:

BTW, thank you for this great convertor and believe that a stable and great finall version is coming ;)

[Milad]

Gold version is coming within few days

I had a test, here is my report:

my site is designed for Chinese-language user. Unfortunately, there are two sets of chinese data with different charsets, utf8 and Big5. I wanted to change all the data to utf8, followed your steps. However, your convertor not only cannot convert the data with charset big5 to utf8, and also change the data with charset utf8 to monster words.

I don't know, I might take some wrong steps :confused:

BTW, thank you for this great convertor and believe that a stable and great finall version is coming ;)

How can you use 2 different charsets in one forum? this is not a good idea. I think.

You have to convert the big5 charset and leave the utf-8 as is.
are the two charsets mixed in your database?

Thank you for feedback :)

[neuston]

How can you use 2 different charsets in one forum? this is not a good idea. I think.
Yes. It is

You have to convert the big5 charset and leave the utf-8 as is.
are the two charsets mixed in your database?
It is the result of importing an old database of xoops. It is my fault to forget that the charset is big5. :tired:

[bmn]

It works fine but i have another problem, i am using utf-8 but while restoring i must use latin2 and when i restored with latin2 and import forum, my İ changes to I and ı changes to i I couldnt fix it with your good program, i tested it and after it converted my chars seems unreadebla. What do you offer?

[Milad]

Converting must be in the place, I don't advise you (for live boards) to convert charset locally and reupload the database again. you may face many trobles with such doing.

Your problem is caused by wrong exporting and importing the database.

Test it only, don't convet your live boards, untill I convert my forum charset and give you the green light.

Thank you

[bmn]

Thanks, i only tested it, not on the live board, then what can i do for this problem, you said that my problem occurs because of the wrong import, how can i do this? If you would help, i will be so happy.

[Milad]

When you choose to convert your forum, you must take a full backup of your database, and then convert it in the place.

To test it well, try to duplicate your database on your live site upload vBulletin files to www.yousite.com/vbtest/ and run it with the duplicated database. and convert this test forum. return here with your feedback.

[Milad]

Beta 2 will come soon with new features.

[dacho]

Milad please help me
I get a Error in charset_converter.php :
Database error in vBulletin 3.5.4:
Invalid SQL:
SELECT * FROM charset_convert WHERE converted=0;
MySQL Error : Table 'dacho_test.charset_convert' doesn't exist
Error Number : 1146
Date : Sunday, July 16th 2006 @ 04:34:00 PM
Script : http://XXX.com/admincp/charset_converter.php
Referrer : http://XXX.com/admincp/index.php?do=nav
IP Address : 255.255.255.255
Username : admin
Classname : vb_database

[Milad]

Milad please help me
I get a Error in charset_converter.php :

You didn't excute the attached SQL file!!!

[dacho]

You didn't excute the attached SQL file!!!
So what did i must to do and how ?

I do all that :
[v]Upload files to their respective locations.
[v]Import charset_convert.sql to your vBulletin database using PHPMyAdmin.
[v]Import the file product product-charset_converter.xml
[v]Refresh your admincp, and go to vCharset Converter->Settings set the Source Charset Encoding and the Destination Charset Encoding.
[v]Go to vBulletin Options -> Censorship Options -> Blank Character Stripper, empty its value and click save.
[/list]

[Milad]

Installation:

Upload files to their respective locations.
Import charset_convert.sql to your vBulletin database using PHPMyAdmin.
Import the file product product-charset_converter.xml
Refresh your admincp, and go to vCharset Converter->Settings set the Source Charset Encoding and the Destination Charset Encoding.
Go to vBulletin Options -> Censorship Options -> Blank Character Stripper, empty its value and click save.


You missed the highlighted line.

[dacho]

You righteous... so i do that

Now i need to convert all the table ?

[Milad]

all the tables in the converter only
and remember

Caution: This is BETA PRODUCT, provided "As Is" without warranties, Test it using a copy of your vBulletin database not using your live vBulletin, Since this is a BETA PRODUCT use it for testing only.

[dacho]

O.K it's work on test forums
In the How to use? [list=1] after the :
" You have to do this with the every choosen tables."
insert : " Go To AdminCP : Language Manager >> Edit Settings >> HTML Character Set."

now i dread to do it on my main forums :(

[dacho]

Beta 2 will come soon with new features.
When ?

[John the Artist]

Worked on vB 3.6 RC2... Awesome.

=)

EDIT:

Why on earth does it blank the custom User titles? Luckily I had them backed up and I'm restoring just that column now. Still, everything else rocks...

[Milad]

Worked on vB 3.6 RC2... Awesome.

=)

EDIT:

Why on earth does it blank the custom User titles? Luckily I had them backed up and I'm restoring just that column now. Still, everything else rocks...
I already said that in the thread, I'm wonder how did it work?

There are some MySQL scheme changes between 3.5.x & 3.6.x

However thank you for your feed back.

[ahmed-samara]

thanks Milad i will test it soon

[FatalBreeze]

thanks! i really needed this one, clicked install! :)

[Australian Gate]

When ?

yah when?

[Milad]

Beta 2 had been ready since two weeks, but I have no time to update the readme.

I hope to be able to post it soon

[Valter]

Hi Milad,

Did you planned to release vB 3.6 version?

Great product and i need it for 3.6.

[Milad]

Yes I planned to, after the releasing of gold versions of both vB3.6 and vChCo(3.5 version).

[Milad]

Beta 2 is released, with new features.

[Khamis]

Dear Milad,

I tried to download this hack into my forum, but it didn't work and I really can't give all the steps I've done because its many. Anyway, I have sent email to you, and I'm waiting for your feedback.

Best Regards,
your brother Abu Mayed

[Milad]

I almostly explained everything.

Do you download the BETA 2?

I replied your email.

[FatalBreeze]

actually, i ran the first beta not the second, and it gave me many problems with the 'word' table.
it just didn't convert it! every time i ran the script on the word table i got an error and was forced to execute the query myself via phpMyAdmin.
but the weird thing was that there was nothing wrong with the sql syntax, because the query worked fine when i ran it throught the phpMyAdmin, and it failed while running through your script.

[Milad]

I advice you to rebiuld the search index.

[Valter]

Beta 2 is released, with new features.
Does it work with 3.6?

Thanks.

[zahrani]

Does it work with 3.6?

Thanks.
I don't think so, it needs more work.

[HaMaDa4eVeR]

My forums encoding is windows-1256, so I can't get full advantages of AJAX functions like the quick reply and the quick edit without some encoding problems.
I Converted all my tables to UTF-8 , but I'm still facing AJAX problems.
thanks :)

[Milad]

Problems like what?

[HaMaDa4eVeR]

I meant the AJAX still not working after I install your hack & convert my forum from windows-1256 to UTF-8
by other meain' I didn't get any worth from your hack....

[Milad]

Did you enable the full AJAX features?

[HaMaDa4eVeR]

yep
and I donot want to disable the problematic AJAX Features,

I want quick repaly and editing to work..

remember my forum encode is utf-8

[Milad]

Can you give me an access to your forum and admincp to see?

[HaMaDa4eVeR]

okay :)
check your PM here

[Milad]

The problem is freezing in the quick reply (not caused by vCharset), you can solve it by add this line to .htaccess

SecFilterScanPOST Off

[HaMaDa4eVeR]

I tried it before but i got internal error page

any way thanks for supporting :(

[warpeditor]

Any update news for 3.6 version?

[anzo]

Hello!

First of all - GREAT HACK! :) I've just installed it on my test 3.6.0 (http://latforums.lv/test/) and it works! The only problems are:
1. It didn't do anything to my shoutbox;
2. I still get strange symbols in error messages.

Is it possible to fix this?

I also couldn't convert the following tables:
cron
phrasetype
post_parsed
profilefield
reputationlevel
subscription
word
threadredirect

[Milad]

This doesn't work for 3.6 the appropriate version hadn't released yet.

It won't ber released before get this as gold version.

[anzo]

well, can't you skip forward to the 3.6 since it's in right now? :) and I mean it works for 3.6, just some minor bugs - that's all.

[warpeditor]

well, can't you skip forward to the 3.6 since it's in right now? :) and I mean it works for 3.6, just some minor bugs - that's all.

please please please!

[warpeditor]

any news? (aka. bump)

[wito]

Any news on 3.6 version?

[j-v-n]

I get

The iconv PHP module isn't presented, please contact your system administrator to enable/install it.

Error but via SSH iconv works like a charm

[Milad]

Any news on 3.6 version?
I need some time to get the difference between 3.5 & 3.6 MySQL schemes.

[Milad]

I get


The iconv PHP module isn't presented, please contact your system administrator to enable/install it.

Error but via SSH iconv works like a charm
look to the php info to see whether this module is installed.

[booktoweb]

I need some time to get the difference between 3.5 & 3.6 MySQL schemes.

please wake up :D

[BonjourChine]

Whats the status of this project regarding 3.6.x ? and is there anyone else with a solution ?

[karabalsagun]

yeah waiting for 3.6.x solution

[Milad]

3.6 version will be released on the first of February.

Please be patient, I'm medical student and I have many exams until 23rd of January.

Regards

[mljask]

did someone try this hack on 3.6.4 ?

[BonjourChine]

3.6 version will be released on the first of February.

Please be patient, I'm medical student and I have many exams until 23rd of January.

Regards

Thanks really looking forward to using it !

[BonjourChine]

3.6 version will be released on the first of February.

Any updated ETA?

[karabalsagun]

3.6 version will be released on the first of February.

Please be patient, I'm medical student and I have many exams until 23rd of January.

Regards

Any Updates ?

[berayiwu5]

3.6 version will be released on the first of February.

Please be patient, I'm medical student and I have many exams until 23rd of January.

Regards

Thanks for working on this. We definitely need it. I hope you can soon come up with a script that works with version 3.6.

[Milad]

Hello

Just to inform you,

vCharset converter 1.0.0 beta 3 for 3.6.x is almost done, and will be released tonight or tomorrow.

Regards

[HaMaDa4eVeR]

Is the vCharset converter 1.0.0 beta 3 working well with vB 3.5.7 ?

[Milad]

No, this branch had been discontinued.

[webramz]

Any update for this mod? I can not find any attachments? Where is the old version?