Converting Special Chars from HTML to UTF-8 ascii standard?

[Kaelon]

Hey there,

I'm using the AddonChat Integration Script and have been working with Chris Duerr, the author, to try and solve this problem: users that have special characters (such as accents, as in ? ? ? ? ?) are getting an invalid username/password notice. This is because vBulletin stores these special characters as HTML escape equivalents.

How can we convert the HTML escape characters to UTF-8 standard ascii characters?

Here is the code cited from the integration script:

<?php
   header("Content-type: text/plain; charset=iso-8859-1");
   error_reporting(E_ALL & ~E_NOTICE);
   define('NO_REGISTER_GLOBALS', 1);
   define('SESSION_BYPASS', 1);
   define('LOCATION_BYPASS', 1);
   //define('DIE_QUIETLY', 1);
    
   /* 
      We lie a little here to let us get through when
      forum read privileges are disabled for non-registered 
      users.
   */
   define('THIS_SCRIPT', 'login');     
   $_REQUEST['do'] = 'register';
   require_once('./global.php');      
   require_once('./chat_global.php');
   
   $username = $_REQUEST['username'];
   $password = $_REQUEST['password'];
   
   /*
      Uncomment the following to support non-ASCII UTF-8 characters
      Requires PHP Multibyte String (mbstring) Extension
   */
   $username = mb_convert_encoding($username, "HTML-ENTITIES", "UTF-8");
   $password = mb_convert_encoding($password, "HTML-ENTITIES", "UTF-8");
   
   
   if(!$SIGMACHAT_VB_AUTHENTICATE) die("DISABLED");
   
   # Fetch User Info from Database..
   $uid = 0;
   if ($userinfo = $db->query_first('SELECT userid, usergroupid, membergroupids, password, salt FROM ' . TABLE_PREFIX . 'user WHERE username = "' . addslashes(htmlspecialchars_uni($username)) . '"'))
   {
      # Invalid Password
    if (($userinfo['password'] != $password) && ($userinfo['password'] != md5(md5($password) . $userinfo['salt'])))    
          $auth = 0;    
    else
    {
   $usergroups = explode(',', $userinfo[membergroupids]);
   $usergroups[] = $userinfo[usergroupid];
         
   $auth = 0;
   foreach($usergroups as $ug)
   {
            if( ($auth < 3) && (in_array($ug, $SIGMACHAT_AUTH_GRANTACCESS)) ) $auth = 3;
   if( ($auth < 2) && (in_array($ug, $SIGMACHAT_AUTH_ADMINACCESS)) ) $auth = 2;
   if( ($auth < 1) && (in_array($ug, $SIGMACHAT_AUTH_ACCESS)) ) $auth = 1;
   if(in_array($ug, $SIGMACHAT_AUTH_NOACCESS)) { $auth = 0; break; }
   }
    $uid = $userinfo['userid'];
    }
   }
   else
   $auth = $SIGMACHAT_AUTH_GUEST;
     
     
   $result_string = "SCRAS^1.1\nAUTH^$auth\nUID^$uid\n";
   
   if($SIGMACHAT_ENABLE_LINK_PROFILE) $result_string .= "SITE_LINK^Profile^$SIGMACHAT_FORUM_URL/chat_func_profile.php\n";
   if($SIGMACHAT_ENABLE_LINK_ADDBUDDY) $result_string .= "SITE_LINK^Add Buddy^$SIGMACHAT_FORUM_URL/chat_func_addbuddy.php\n";   
   if($SIGMACHAT_ENABLE_LINK_PM) $result_string .= "SITE_LINK^Prv. Message^$SIGMACHAT_FORUM_URL/chat_func_pm.php\n";
   if($SIGMACHAT_ENABLE_LINK_EMAIL) $result_string .= "SITE_LINK^eMail^$SIGMACHAT_FORUM_URL/chat_func_email.php\n";   
   if($SIGMACHAT_ENABLE_LINK_FINDPOSTS) $result_string .= "SITE_LINK^Find Posts^$SIGMACHAT_FORUM_URL/chat_func_findposts.php\n";   
   if($SIGMACHAT_ENABLE_LINK_FORUM_IGNORE) $result_string .= "SITE_LINK^Forum Ignore^$SIGMACHAT_FORUM_URL/chat_func_ignore.php\n";      
   
   print($result_string);   
   
?>

Update -- I've tried using html_entity_decode by calling as follows:

$username = html_entity_decode($username);
$password = html_entity_decode($password);

... where the "uncomment the following" comment is indicated in the above code. That didn't work, tragically.

[Paul M]

There is a function in vb called unhtmlspecialchars()

From the documentation ;

Returns a string where HTML entities have been converted back to their original characters

string unhtmlspecialchars (string $text, [boolean $doUniCode = false]) 

string $text: String to be parsed 

boolean $doUniCode: Convert unicode characters back from HTML entities?

[Kaelon]

Thanks, Paul! However, that didn't seem to work. I added:

	$username = unhtmlspecialchars($username);
	$password = unhtmlspecialchars($password);

... to the previous mb_convert_encoding command-lines, and I was still getting invalid returns from the system. Judging by the code above, is there a more sensible place to convert the unhtmlspecialchars to validate this? Thanks!

[Kaelon]

Latest information from Chris Duerr, the original hack author:

Originally Posted by cduerr

I'm not familiar with that command -- but it almost seems like you'd want to do the reverse; that is convert the special chars to their HTML representation. Sometimes function names can be confusing though, so you may have the right function.

Do you know the usage of the command, ideally it would be a drop-in replacement for the mb_convert_encoding commands -- it'll be one of the first commands you run in the script.

What we typically do when debugging this sort of thing is to write the output data to a text file (using php file commands within the authentication script) as there is no easy way to simply echo the information to the console when using special characters. This may help by first printing the raw data we send, then print the data as you've converted it, and finally print the raw data stored in the database for comparison to gauge your progress.

Accordingly, is the opposite of unhtmlspecialchars() just htmlspecialchars()?

[Paul M]

I didn't really read your code, you asked about decoding, which was what I answered.

Looking at your code then yes, you need to do the opposite, you want to code your username to match vb. The vb function is htmlspecialchars_uni(), but I believe vb does more than just that.

[Kaelon]

Thanks, Paul. I gave that a shot, but strangely, still no luck. Specifically, I used:

$username = htmlspecialchars_uni($username);
$password = htmlspecialchars_uni($password);

... and I still got invalid returns from the system. Then looking further, I also saw that the chat_auth.php code provided by Chris Duerr had already apparently done this analysis:

   # Fetch User Info from Database..
   $uid = 0;
   if ($userinfo = $db->query_first('SELECT userid, usergroupid, membergroupids, password, salt FROM ' . TABLE_PREFIX . 'user WHERE username = "' . addslashes(htmlspecialchars_uni($username)) . '"'))
   {
      # Invalid Password
    if (($userinfo['password'] != $password) && ($userinfo['password'] != md5(md5($password) . $userinfo['salt'])))    
          $auth = 0;    
    else
...

[Paul M]

You need to look in the user datamanager to see what other conversions vb does.

[Kaelon]

Originally Posted by Paul M

You need to look in the user datamanager to see what other conversions vb does.

Sounds good. Where can I find the user datamanager?

[Paul M]

class_dm_user.php in the includes folder.

[Grim77]

Kaelon -- Just curious if we ever found a solution to this? I'm working on the 3.7 mod now, and would like to find a solution that doesn't require a non-standard php library.