vb.org Archive - Board Optimization - vCharset Converter 1.0.0 BETA 2

vb.org Archive (https://vborg.vbsupport.ru/index.php)

- Modification Graveyard (https://vborg.vbsupport.ru/forumdisplay.php?f=224)

- - Board Optimization - vCharset Converter 1.0.0 BETA 2 (https://vborg.vbsupport.ru/showthread.php?t=118308)

vCharset Converter 1.0.0 BETA 2

THE 3.5 BRANCHE IS CLOSED, A NEW VERSION FOR 3.6 HAD BEEN RELEASED.

To use this modification, please update to 3.6.x and use this.

Quote:

By syrianmeds.net

Description: This script converts the database of vBulletin forums (from and to) any of the charsets that are supported by iconv. also it has the ability to convert the tables of the hacks you had installed them.

Caution: This is BETA PRODUCT, provided "As Is" without warranties, Test it using a copy of your vBulletin database not using your live vBulletin, Since this is a BETA PRODUCT use it for testing only.

Why this product?
My forums encoding is windows-1256, so I can't get full advantages of AJAX functions like the quick reply and the quick edit without some encoding problems.
I decided to convert my encoding and build this script. and I'll convert to utf-8 (Done).
This product was tested on a copy of real live vBulletin and it seems to be good.

Requirements:

vBulletin 3.5.4+
iconv php extension is installed on your system, or your PHP version is PHP5

Supported Encodings:
See http://www.gnu.org/software/libiconv/#introduction

Installation:

Upload files to their respective locations.
Import the file product product-charset_converter.xml (Choose Yes when Upgrade)
Refresh your admincp, go to vCharset Converter->Settings and set (the Source Charset Encoding & the Destination Charset Encoding).
Go to vBulletin Options -> Censorship Options -> Blank Character Stripper, empty its value and click save.

How to use?

Close your forum.
Backup your database.
Add a new language and call it English for example, its charset is your Destination Charset Encoding.
Set your options (profile.php?do=editoptions) to use it as a default language.
Change the charset of your old langauge into Destination Charset Encoding.
Go to vCharset Converter -> Manage tables. I'll leave this to you to explore it. But edit the userfield table and the custom fields to it.
Go to vCharset Converter -> vCharset Converter. if you see a message like:

Quote:

The iconv PHP module is not presented, please contact your system administrator to enable/install it.

this means you need to install iconv.
You will see a drop-down menu. select the table you want to convert, set the number of items to convert per cycle (default 1000) and press submit.
You have to do this with each table in the drop-down menu.
Test your forum search for Non-English letters, if you don't get the expected results, read the "Search Problems".
Open your forum.
After finishing tell me what are your impression, idea and faced problems.

Search Problems
What is the problem? After converting you may get no (well-expected) results when doing search, I had faced this problem after converting to UTF-8.
What is my circumstances? Mysql 4.1.x, Full-Text search and I converted to utf-8.
What is the reason? Full-Text search has Restrictions that say: "all columns in a FULLTEXT index must use the same character set and collation.".
So if you use fulltext search with MySQL 4.1.x and convert to utf-8 perform these queries: (after converting post and thread tables)

Code:

ALTER TABLE post MODIFY pagetext mediumtext COLLATE utf8_unicode_ci

ALTER TABLE post MODIFY title VARCHAR(250) COLLATE utf8_unicode_ci

ALTER TABLE thread MODIFY title VARCHAR(250) COLLATE utf8_unicode_ci

Dont forget to add your table prefix if you have

- If you use vBulletin built-in search engine (with any MySQL version), then consider to Rebuild Search Index (Maintenance -> Update Counters -> Rebuild Search Index)
You may read this thread http://www.vbulletin.com/forum/showthread.php?t=191315

Email Problems
After the conversion, my users had reported that they couldn't read the notification emails. the emails are sent with utf-8 encoding.
Not all the users do know how to change the encoding of a page. you may learn them, this is the only solution.
Problem had been faced with Hotmail (confirmed by me) and Yahoo (unconfirmed).
No problem with Gmail (confirmed by me).

Known issues:

Users titles had gone away after converting. (Still exists in beta 2)

Versions

1.0.0 BETA 1 (initial release). (11. Jun 2006)
1.0.0 BETA 2 new features: (4. Aug 2006)
1. Manage Tables: Now you can add or edit your tables, this will make converting of additional tables very easy.
2. New built-in images to mark the tables:
  - Attachment 50952 Table is ready to be converted.
  - Attachment 50955 Table is uknown.
  - Attachment 50954 No need to convert this table.
  - Attachment 50953 Table Converted :)
3. Now you can see all the tables in your database, with their statues and you have the ability to change this status to "no need" or you can add the fields and the primaryid field, or you can edit the information of any table before you go on converting it.

Forums had been Converted using this tool: (PM me to list your forum)

http://www.basharialep.com/forum/ from windows-1256 into UTF-8

Support is given for who clicked "install" only.

Click https://vborg.vbsupport.ru/ to recieve updates.

Nominate it for Mod o.t. Month if you love it.

thanks Milad i will test it soon

Your script is extremely good. During the last 4 days I am trying to convert the database of my forum from greek ISO to utf8, I tried hundrend ways without the desired output and when I installed the vCharset all conversions made automagically and without any problems as fas as I can see now.

I have a question: I removed a part of data of my database by accident and now in product list (in admincp) vCharset is not shown anymore (in order to uninstall it since the convert is done). Also menu "vCharset" in admincp is not shown anymore. But I've got a table "charset_convert" in my database so I assume there are some parts of vCharset installtion still in forum.

Is there a sql command that I can run in order to remove any part of vCharset in my forum?

Thanks for your feedback

I assume you ran the script on a test board not on live one.

vCharset converter must remain in your database after finishing, it doesn't uninstal itself, you can uninstall it manually. yes you can uninstall the product, drop the table charset_convert and delete the product files.

Because you delete something in your database, you don't see the product, I think you deleted or epmtied product table.

I advise you to return to your backup, and test my script many times before you make decision to go ahead with it.

Remember I am still considering this product as a BETA product.

Thanks for your feedback again. :)

Also if you have some hacks, extensions or plugins that collect some text information in the database, you have to convert them too, this will be explained after the gold is released.

thanks alot Milad
thats extremely good

Go on developing this
we need to see a strong stable version as soon as posible

;)

Thanks

Please to keep yourselves updated press install.

Quote:

Originally Posted by Milad

is there any reason that the convertor must stay after finishing the convertion?
thanks!

Quote:

Originally Posted by blacklude

is there any reason that the convertor must stay after finishing the convertion?

Never

But i want you to know that the product doesn't uninstall itself after the converting, you should uninstall it yourself.

Regards

Any ideas? suggestions? or bugs?

ميلاد جربته لكن طلع خطا في اكثر من جدول والتجربة كانت على 3.6.0
وبعدين لما اعرض السورس من المصدر تطلعلي جمل غريبة
ط?ظ?ظ?ظ?ط?ط?ظٹط?: 1, ط?ظ?ظ?ط?ط?ط?ظ?ط?طھ: 5, ط?ظ?ط?ط?ط?ط?ط،: 1<
وبعدين الرد السريع ما يرسل اضغط عليه يرجع مكانه لما اسوي رفرش يطلع الرد بالموضوع
وهذه صورة بالمرفق للجداول الي ما صارت او ما تحولت
اعتقد يمكن من اصدارة النسخة البيتا الجديدة
عندي منتدى ضخم بجرب عليه اصداره 3.4.5
وبردلك خبر

thanks :) :banana:

This is for vB3.5.x only, for the source code it's normal to see it like ': 5, ط?ظ„ط?ط?ط?ط'

also do you change the charset html for the languages into utf-8? I think you didn't.

Please don't use it on live boards, and only for 3.5.x

Thanks for your feedback, and I'm waiting for more :)

Great work mr. Milad i'll test it in forums ..

be right back .. ..

I hope you'll release the gold version soon.

I had a test, here is my report:

my site is designed for Chinese-language user. Unfortunately, there are two sets of chinese data with different charsets, utf8 and Big5. I wanted to change all the data to utf8, followed your steps. However, your convertor not only cannot convert the data with charset big5 to utf8, and also change the data with charset utf8 to monster words.

I don't know, I might take some wrong steps :confused:

BTW, thank you for this great convertor and believe that a stable and great finall version is coming ;)

Gold version is coming within few days

Quote:

Originally Posted by neuston

How can you use 2 different charsets in one forum? this is not a good idea. I think.

You have to convert the big5 charset and leave the utf-8 as is.
are the two charsets mixed in your database?

Thank you for feedback :)

Quote:

Originally Posted by Milad

How can you use 2 different charsets in one forum? this is not a good idea. I think.

Yes. It is

Quote:

Originally Posted by Milad

You have to convert the big5 charset and leave the utf-8 as is.
are the two charsets mixed in your database?

It is the result of importing an old database of xoops. It is my fault to forget that the charset is big5. :tired:

It works fine but i have another problem, i am using utf-8 but while restoring i must use latin2 and when i restored with latin2 and import forum, my İ changes to I and ı changes to i I couldnt fix it with your good program, i tested it and after it converted my chars seems unreadebla. What do you offer?

Converting must be in the place, I don't advise you (for live boards) to convert charset locally and reupload the database again. you may face many trobles with such doing.

Your problem is caused by wrong exporting and importing the database.

Test it only, don't convet your live boards, untill I convert my forum charset and give you the green light.

Thank you

Thanks, i only tested it, not on the live board, then what can i do for this problem, you said that my problem occurs because of the wrong import, how can i do this? If you would help, i will be so happy.

When you choose to convert your forum, you must take a full backup of your database, and then convert it in the place.

To test it well, try to duplicate your database on your live site upload vBulletin files to www.yousite.com/vbtest/ and run it with the duplicated database. and convert this test forum. return here with your feedback.

Beta 2 will come soon with new features.

Milad please help me
I get a Error in charset_converter.php :

Quote:

Database error in vBulletin 3.5.4:
Invalid SQL:
SELECT * FROM charset_convert WHERE converted=0;
MySQL Error : Table 'dacho_test.charset_convert' doesn't exist
Error Number : 1146
Date : Sunday, July 16th 2006 @ 04:34:00 PM
Script : http://XXX.com/admincp/charset_converter.php
Referrer : http://XXX.com/admincp/index.php?do=nav
IP Address : 255.255.255.255
Username : admin
Classname : vb_database

Quote:

Originally Posted by dacho

Milad please help me
I get a Error in charset_converter.php :

You didn't excute the attached SQL file!!!

Quote:

Originally Posted by Milad

You didn't excute the attached SQL file!!!

So what did i must to do and how ?

I do all that :
[v]Upload files to their respective locations.
[v]Import charset_convert.sql to your vBulletin database using PHPMyAdmin.
[v]Import the file product product-charset_converter.xml
[v]Refresh your admincp, and go to vCharset Converter->Settings set the Source Charset Encoding and the Destination Charset Encoding.
[v]Go to vBulletin Options -> Censorship Options -> Blank Character Stripper, empty its value and click save.[/list]

Quote:

Originally Posted by me

Installation:

Upload files to their respective locations.
Import charset_convert.sql to your vBulletin database using PHPMyAdmin.
Import the file product product-charset_converter.xml
Refresh your admincp, and go to vCharset Converter->Settings set the Source Charset Encoding and the Destination Charset Encoding.
Go to vBulletin Options -> Censorship Options -> Blank Character Stripper, empty its value and click save.

You missed the highlighted line.

You righteous... so i do that

Now i need to convert all the table ?

all the tables in the converter only
and remember

Quote:

Caution: This is BETA PRODUCT, provided "As Is" without warranties, Test it using a copy of your vBulletin database not using your live vBulletin, Since this is a BETA PRODUCT use it for testing only.

O.K it's work on test forums
In the How to use?[list=1] after the :
"[*]You have to do this with the every choosen tables."
insert : "[*]Go To AdminCP : Language Manager >> Edit Settings >> HTML Character Set."

now i dread to do it on my main forums :(

Quote:

Originally Posted by Milad

Beta 2 will come soon with new features.

When ?

Worked on vB 3.6 RC2... Awesome.

=)

EDIT:

Why on earth does it blank the custom User titles? Luckily I had them backed up and I'm restoring just that column now. Still, everything else rocks...

Quote:

Originally Posted by John the Artist

Worked on vB 3.6 RC2... Awesome.

=)

EDIT:

Why on earth does it blank the custom User titles? Luckily I had them backed up and I'm restoring just that column now. Still, everything else rocks...

I already said that in the thread, I'm wonder how did it work?

There are some MySQL scheme changes between 3.5.x & 3.6.x

However thank you for your feed back.

thanks Milad i will test it soon

thanks! i really needed this one, clicked install! :)

Quote:

Originally Posted by dacho

When ?

yah when?

Beta 2 had been ready since two weeks, but I have no time to update the readme.

I hope to be able to post it soon

Hi Milad,

Did you planned to release vB 3.6 version?

Great product and i need it for 3.6.

Yes I planned to, after the releasing of gold versions of both vB3.6 and vChCo(3.5 version).

Beta 2 is released, with new features.

Phrase Groups Available:

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.01498 seconds Memory Usage 1,853KB Queries Executed 10 (?)
More Information
Template Usage: (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (1)bbcode_code_printable (15)bbcode_quote_printable (1)footer (1)gobutton (1)header (1)headinclude (6)option (1)pagenav (1)pagenav_curpage (1)pagenav_pagelink (1)post_thanks_navbar_search (1)printthread (40)printthreadbit (1)spacer_close (1)spacer_open Phrase Groups Available: global postbit showthread	Included Files: ./printthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/class_bbcode_alt.php ./includes/class_bbcode.php ./includes/functions_bigthree.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete printthread_start pagenav_page pagenav_complete bbcode_fetch_tags bbcode_create bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete printthread_post printthread_complete
Messages: