Board Optimization - vCharset Converter 1.0.0 BETA 2

THE 3.5 BRANCHE IS CLOSED, A NEW VERSION FOR 3.6 HAD BEEN RELEASED.

To use this modification, please update to 3.6.x and use this.

By syrianmeds.net

Description: This script converts the database of vBulletin forums (from and to) any of the charsets that are supported by iconv. also it has the ability to convert the tables of the hacks you had installed them.

Caution: This is BETA PRODUCT, provided "As Is" without warranties, Test it using a copy of your vBulletin database not using your live vBulletin, Since this is a BETA PRODUCT use it for testing only.

Why this product?
My forums encoding is windows-1256, so I can't get full advantages of AJAX functions like the quick reply and the quick edit without some encoding problems.
I decided to convert my encoding and build this script. and I'll convert to utf-8 (Done).
This product was tested on a copy of real live vBulletin and it seems to be good.

Requirements:

• vBulletin 3.5.4+
• iconv php extension is installed on your system, or your PHP version is PHP5

Supported Encodings:
See http://www.gnu.org/software/libiconv/#introduction

Installation:

1. Upload files to their respective locations.
2. Import the file product product-charset_converter.xml (Choose Yes when Upgrade)
3. Refresh your admincp, go to vCharset Converter->Settings and set (the Source Charset Encoding & the Destination Charset Encoding).
4. Go to vBulletin Options -> Censorship Options -> Blank Character Stripper, empty its value and click save.

How to use?

1. Close your forum.
2. Backup your database.
3. Add a new language and call it English for example, its charset is your Destination Charset Encoding.
4. Set your options (profile.php?do=editoptions) to use it as a default language.
5. Change the charset of your old langauge into Destination Charset Encoding.
6. Go to vCharset Converter -> Manage tables. I'll leave this to you to explore it. But edit the userfield table and the custom fields to it.
7. Go to vCharset Converter -> vCharset Converter. if you see a message like:
   this means you need to install iconv.
8. You will see a drop-down menu. select the table you want to convert, set the number of items to convert per cycle (default 1000) and press submit.
9. You have to do this with each table in the drop-down menu.
10. Test your forum search for Non-English letters, if you don't get the expected results, read the "Search Problems".
11. Open your forum.
12. After finishing tell me what are your impression, idea and faced problems.

Search Problems
What is the problem? After converting you may get no (well-expected) results when doing search, I had faced this problem after converting to UTF-8.
What is my circumstances? Mysql 4.1.x, Full-Text search and I converted to utf-8.
What is the reason? Full-Text search has Restrictions that say: "all columns in a FULLTEXT index must use the same character set and collation.".
So if you use fulltext search with MySQL 4.1.x and convert to utf-8 perform these queries: (after converting post and thread tables)

ALTER TABLE post MODIFY pagetext mediumtext COLLATE utf8_unicode_ci
ALTER TABLE post MODIFY title VARCHAR(250) COLLATE utf8_unicode_ci
ALTER TABLE thread MODIFY title VARCHAR(250) COLLATE utf8_unicode_ci

Dont forget to add your table prefix if you have

- If you use vBulletin built-in search engine (with any MySQL version), then consider to Rebuild Search Index (Maintenance -> Update Counters -> Rebuild Search Index)
You may read this thread http://www.vbulletin.com/forum/showthread.php?t=191315

Email Problems
After the conversion, my users had reported that they couldn't read the notification emails. the emails are sent with utf-8 encoding.
Not all the users do know how to change the encoding of a page. you may learn them, this is the only solution.
Problem had been faced with Hotmail (confirmed by me) and Yahoo (unconfirmed).
No problem with Gmail (confirmed by me).

Known issues:

1. Users titles had gone away after converting. (Still exists in beta 2)

Versions

• 1.0.0 BETA 1 (initial release). (11. Jun 2006)
• 1.0.0 BETA 2 new features: (4. Aug 2006)
  1. Manage Tables: Now you can add or edit your tables, this will make converting of additional tables very easy.
  2. New built-in images to mark the tables:
     • Attachment 50952 Table is ready to be converted.
     • Attachment 50955 Table is uknown.
     • Attachment 50954 No need to convert this table.
     • Attachment 50953 Table Converted
  3. Now you can see all the tables in your database, with their statues and you have the ability to change this status to "no need" or you can add the fields and the primaryid field, or you can edit the information of any table before you go on converting it.

Forums had been Converted using this tool: (PM me to list your forum)

• http://www.basharialep.com/forum/ from windows-1256 into UTF-8

Support is given for who clicked "install" only.

Click to recieve updates.

Nominate it for Mod o.t. Month if you love it.

[Milad]

Any ideas? suggestions? or bugs?

[e7lew]

ميلاد جربته لكن طلع خطا في اكثر من جدول والتجربة كانت على 3.6.0
وبعدين لما اعرض السورس من المصدر تطلعلي جمل غريبة
ط?ظ?ظ?ظ?ط?ط?ظٹط?: 1, ط?ظ?ظ?ط?ط?ط?ظ?ط?طھ: 5, ط?ظ?ط?ط?ط?ط?ط،: 1<
وبعدين الرد السريع ما يرسل اضغط عليه يرجع مكانه لما اسوي رفرش يطلع الرد بالموضوع
وهذه صورة بالمرفق للجداول الي ما صارت او ما تحولت
اعتقد يمكن من اصدارة النسخة البيتا الجديدة
عندي منتدى ضخم بجرب عليه اصداره 3.4.5
وبردلك خبر

thanks :banana:

[Milad]

This is for vB3.5.x only, for the source code it's normal to see it like ': 5, ط?ظ„ط?ط?ط?ط'

also do you change the charset html for the languages into utf-8? I think you didn't.

Please don't use it on live boards, and only for 3.5.x

Thanks for your feedback, and I'm waiting for more

[Almotmaiz.Net]

Great work mr. Milad i'll test it in forums ..

be right back .. ..

[Lizard King]

I hope you'll release the gold version soon.

[neuston]

I had a test, here is my report:

my site is designed for Chinese-language user. Unfortunately, there are two sets of chinese data with different charsets, utf8 and Big5. I wanted to change all the data to utf8, followed your steps. However, your convertor not only cannot convert the data with charset big5 to utf8, and also change the data with charset utf8 to monster words.

I don't know, I might take some wrong steps

BTW, thank you for this great convertor and believe that a stable and great finall version is coming

[Milad]

Gold version is coming within few days

Originally Posted by neuston

I had a test, here is my report:

my site is designed for Chinese-language user. Unfortunately, there are two sets of chinese data with different charsets, utf8 and Big5. I wanted to change all the data to utf8, followed your steps. However, your convertor not only cannot convert the data with charset big5 to utf8, and also change the data with charset utf8 to monster words.

I don't know, I might take some wrong steps

BTW, thank you for this great convertor and believe that a stable and great finall version is coming

How can you use 2 different charsets in one forum? this is not a good idea. I think.

You have to convert the big5 charset and leave the utf-8 as is.
are the two charsets mixed in your database?

Thank you for feedback

[neuston]

Originally Posted by Milad

How can you use 2 different charsets in one forum? this is not a good idea. I think.

Yes. It is

Originally Posted by Milad

You have to convert the big5 charset and leave the utf-8 as is.
are the two charsets mixed in your database?

It is the result of importing an old database of xoops. It is my fault to forget that the charset is big5. :tired:

[bmn]

It works fine but i have another problem, i am using utf-8 but while restoring i must use latin2 and when i restored with latin2 and import forum, my İ changes to I and ı changes to i I couldnt fix it with your good program, i tested it and after it converted my chars seems unreadebla. What do you offer?

[Milad]

Converting must be in the place, I don't advise you (for live boards) to convert charset locally and reupload the database again. you may face many trobles with such doing.

Your problem is caused by wrong exporting and importing the database.

Test it only, don't convet your live boards, untill I convert my forum charset and give you the green light.

Thank you