Go Back   vb.org Archive > vBulletin Modifications > Archive > Modification Graveyard
FAQ Community Calendar Today's Posts Search

Reply
 
Thread Tools
vB Global Translator - Multiply your indexed pages & put search traffic on autopilot Details »»
vB Global Translator - Multiply your indexed pages & put search traffic on autopilot
Version: 2.4, by Dave Hybrid Dave Hybrid is offline
Developer Last Online: Dec 2013 Show Printable Version Email this Page

Category: Miscellaneous Hacks - Version: 3.8.x Rating:
Released: 06-27-2009 Last Update: 07-07-2009 Installs: 67
Uses Plugins Template Edits
Additional Files  
No support by the author.

**Text Removed**

Show Your Support

  • This modification may not be copied, reproduced or published elsewhere without author's permission.

Comments
  #252  
Old 07-05-2009, 11:15 AM
Dave Hybrid's Avatar
Dave Hybrid Dave Hybrid is offline
 
Join Date: Mar 2007
Posts: 463
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Ok, but i ran your duplication update a few days ago, i have to run it again for this update too?
Reply With Quote
  #253  
Old 07-05-2009, 11:50 AM
NLP-er's Avatar
NLP-er NLP-er is offline
 
Join Date: Aug 2008
Location: Wrocław
Posts: 1,353
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally Posted by Dave Hybrid View Post
Ok, but i ran your duplication update a few days ago, i have to run it again for this update too?
YES! As I wrote - before my update it is possible that data are duplicated, when some pages are translated in same time. spiders usually works with friends So since your last removal you have again duplication. It will not happen again after this update (ok it will but only for table wt_cache - it's mysql limitation, but tables with biggest amount of data will not have duplicated data again).
Reply With Quote
  #254  
Old 07-05-2009, 12:59 PM
Dave Hybrid's Avatar
Dave Hybrid Dave Hybrid is offline
 
Join Date: Mar 2007
Posts: 463
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Understood, i'll try again in a bit and then update the main install. Thanks.
Reply With Quote
  #255  
Old 07-05-2009, 04:36 PM
jaryx's Avatar
jaryx jaryx is offline
 
Join Date: May 2007
Posts: 42
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Does anyone use vBseo Relevant Replacement?

RR also be translated? Unfortunately, not on my forum :/
Reply With Quote
  #256  
Old 07-05-2009, 07:40 PM
mike2902's Avatar
mike2902 mike2902 is offline
 
Join Date: Apr 2009
Posts: 186
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Can someone help me understand this. The goal of this MOD is to get a flood of international traffic. This traffic is unlikely to stick around because of the language issue so whats the point of wanting all this traffic.
Reply With Quote
  #257  
Old 07-05-2009, 08:30 PM
Dave Hybrid's Avatar
Dave Hybrid Dave Hybrid is offline
 
Join Date: Mar 2007
Posts: 463
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally Posted by mike2902 View Post
Can someone help me understand this. The goal of this MOD is to get a flood of international traffic. This traffic is unlikely to stick around because of the language issue so whats the point of wanting all this traffic.
Same reason you want normal traffic, not everyone wants to stay and chat with them either.

For me I use it for ad revenue, branding, future referrals from word of mouth to UK traffic, it's free exposure.

Your mileage may vary.
Reply With Quote
  #258  
Old 07-05-2009, 08:50 PM
Dave Hybrid's Avatar
Dave Hybrid Dave Hybrid is offline
 
Join Date: Mar 2007
Posts: 463
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally Posted by NLP-er View Post
Hello. Just worked out for you another data base update which will made it without data duplication and also faster

So really on one table still you can have duplicated data, because of mysql unique limitations I didn't set it in wt_cache, but this table has less data than other updated tables.

Also collate was changed - believe it works faster when makes collation by binary values, that by some additional rules like in previous DB.

To have faster DB without data duplication You have to.

For new installations
Here is new DB:
Code:
CREATE TABLE wt_cache (
id INT AUTO_INCREMENT NOT NULL PRIMARY KEY,
tl VARCHAR(10),
originaltext VARCHAR(65000),
translated TEXT,
INDEX(originaltext(50), tl)
) ENGINE = MYISAM, CHARACTER SET utf8 COLLATE utf8_bin;

CREATE TABLE wt_cache_medium (
id INT AUTO_INCREMENT NOT NULL PRIMARY KEY,
tl VARCHAR(10),
originaltext VARCHAR(255),
translated VARCHAR(1000),
UNIQUE (originaltext, tl)
) ENGINE = MYISAM, CHARACTER SET utf8 COLLATE utf8_bin;

CREATE TABLE wt_cache_short (
id INT AUTO_INCREMENT NOT NULL PRIMARY KEY,
tl VARCHAR(10),
originaltext VARCHAR(50),
translated VARCHAR(255),
UNIQUE (originaltext, tl)
) ENGINE = MYISAM, CHARACTER SET utf8 COLLATE utf8_bin;
Also change code in translate.php
this one:
Code:
/* Check cache for translation */
if ($enablecache) {
  $sql=null;
  $length = strlen($text);
  if ($length<=50) {
    $sql = mysql_query("SELECT translated FROM wt_cache_short WHERE originaltext='".addslashes($text)."' AND tl='".addslashes($tl)."' LIMIT 1");  
  } else if ($length > 255) {
    $sql = mysql_query("SELECT translated FROM wt_cache WHERE originaltext='".addslashes($text)."' AND tl='".addslashes($tl)."' LIMIT 1");
  } else {
    $sql = mysql_query("SELECT translated FROM wt_cache_medium WHERE originaltext='".addslashes($text)."' AND tl='".addslashes($tl)."' LIMIT 1"); 
  }
  
	while($t = mysql_fetch_array($sql)) {
		return $lsto.$t['translated'].$rsto;
	}
}
/* -- if not found, proceed with Google Translate */
To this one:
Code:
/* Check cache for translation */
if ($enablecache) {
  $sql=null;
  $length = strlen($text);

  if ($length<=50) {
    $sql = mysql_query("SELECT translated FROM wt_cache_short WHERE originaltext='".addslashes($text)."' AND tl='".addslashes($tl)."'") or die('Query to short cache failed: ' . mysql_error());
  } else if ($length > 255) {
    $sql = mysql_query("SELECT translated FROM wt_cache WHERE originaltext='".addslashes($text)."' AND tl='".addslashes($tl)."' LIMIT 1") or die('Query to normal cache failed: ' . mysql_error());
  } else {
    $sql = mysql_query("SELECT translated FROM wt_cache_medium WHERE originaltext='".addslashes($text)."' AND tl='".addslashes($tl)."'") or die('Query to medium cache failed: ' . mysql_error());
  }
  
	while($t = mysql_fetch_array($sql)) {
		return $lsto.$t['translated'].$rsto;
	}
}
/* -- if not found, proceed with Google Translate */
And change this one:
Code:
////////////////////////////////////////////////////
if ($enablecache) {
mysql_connect ($mysqlserver, $dbusername, $dbpassword);
mysql_select_db ($dbname);
mysql_set_charset('utf8');
}
////////////////////////////////////////////////////
To this one (less white screns):
Code:
////////////////////////////////////////////////////
if ($enablecache) {
  establishConnection();
}


function establishConnection() {
  global $dbusername, $dbpassword, $mysqlserver, $dbname;
  mysql_connect ($mysqlserver, $dbusername, $dbpassword) or die('Could not connect: ' . mysql_error());
  mysql_select_db ($dbname) or die('Could not select database');
  mysql_set_charset('utf8');
}
////////////////////////////////////////////////////
And last change. This one:
Code:
/* Save to cache */
if ($enablecache && (strlen($ttext)>0)) {
  $length = strlen($text);
  if ($length<=50) {
    mysql_query("INSERT INTO wt_cache_short SET tl='".addslashes($tl)."', originaltext='".addslashes($text)."', translated='".addslashes(trim($ttext))."'");
  } else if ($length > 255) {
    mysql_query("INSERT INTO wt_cache SET tl='".addslashes($tl)."', originaltext='".addslashes($text)."', translated='".addslashes(trim($ttext))."'");
  } else {
    mysql_query("INSERT INTO wt_cache_medium SET tl='".addslashes($tl)."', originaltext='".addslashes($text)."', translated='".addslashes(trim($ttext))."'");
  }
}
/* -- */
To this one:
Code:
/* Save to cache */
if ($enablecache && (strlen($ttext)>0)) {
  $length = strlen($text);

  if (!mysql_ping()) {
    mysql_close();
    establishConnection();
  }
  
  if ($length<=50) {
    mysql_query("INSERT IGNORE INTO wt_cache_short SET tl='".addslashes($tl)."', originaltext='".addslashes($text)."', translated='".addslashes(trim($ttext))."'") or die('Insert to short cache failed: ' . mysql_error());
  } else if ($length > 255) {
    mysql_query("INSERT INTO wt_cache SET tl='".addslashes($tl)."', originaltext='".addslashes($text)."', translated='".addslashes(trim($ttext))."'") or die('Insert to normal cache failed: ' . mysql_error());
  } else {
    mysql_query("INSERT IGNORE INTO wt_cache_medium SET tl='".addslashes($tl)."', originaltext='".addslashes($text)."', translated='".addslashes(trim($ttext))."'") or die('Insert to medium cache failed: ' . mysql_error());
  }
}
/* -- */
So whole new translate.php is:
Code:
<?php
////////////////////////////////////////////////////
// Global Translator API
////////////////////////////////////////////////////

global $enablesession, $enablecache, $originalEncoding, $fl;
global $dbusername, $dbpassword, $mysqlserver, $dbname;

////////////////////////////////////////////////////
//        SETTINGS
////////////////////////////////////////////////////
$enablesession = false; //ignore
$enablecache = true; //true - enable cache setting, false - disable cache setting
$originalEncoding = 'iso-8859-1'; //refer to your forum source code for your base encoding
$fl = 'en'; //current language (original) - refer table for language variables

$dbusername = "SET_IT";
$dbpassword = "SET_IT";
$mysqlserver = "localhost";
$dbname = "SET_IT";

////////////////////////////////////////////////////

if ($enablesession) {
        @session_name("iPWTLang");
        @session_start();
}


set_time_limit(0);


////////////////////////////////////////////////////
if ($enablecache) {
  establishConnection();
}


function establishConnection() {
  global $dbusername, $dbpassword, $mysqlserver, $dbname;
  mysql_connect ($mysqlserver, $dbusername, $dbpassword) or die('Could not connect: ' . mysql_error());
  mysql_select_db ($dbname) or die('Could not select database');
  mysql_set_charset('utf8');
}
////////////////////////////////////////////////////

function api_strip ($dt) { $dt = str_replace("\\\\", '\\', $dt); return ($dt); }

function emodfix ($dt) { $dt = str_replace('\"', '"', $dt); return ($dt); }

function remove_tags ($html) {
/* Remove reserved triple tags in html */
return preg_replace ("|<<<([^><]*)>>>|e", "emodfix('\\1')", $html);
}

function add_tags ($buffer) {
/* Inject triple tags to html to preserve html content from translation using ob_start */
return preg_replace("/(^|>\/?)([^><]*)($|\/?<)/e",
             "emodfix('\\1').'<<<'.emodfix('\\2').'>>>'.emodfix('\\3')",
             $buffer);
}

function getServerLoad($windows=false)
{
//MY CODE TO DISABLE SERVER LOAD CHEcKING
  return 1;
//END

    $os = strtolower(PHP_OS);
    if (strpos($os, "win") === false) {
        if (file_exists("/proc/loadavg")) {
            $data = file_get_contents("/proc/loadavg");
            $load = explode(' ', $data);
            return $load[0];

        } elseif (function_exists("shell_exec")) {

            $load = explode(' ', `uptime`);
            return $load[count($load)-1];

        } else {

            return false;
        }

    } elseif($windows) {

        if(class_exists("COM")) {
            $wmi = new COM("WinMgmts:\\\\.");
            $cpus = $wmi->InstancesOf("Win32_Processor");

             $cpuload = 0;
             $i = 0;

            if(version_compare('4.50.0', PHP_VERSION) == 1) {
                // PHP 4
                while ($cpu = $cpus->Next()) {
                    $cpuload += $cpu->LoadPercentage;
                    $i++;
                }

            } else {

                // PHP 5
                foreach ( $cpus as $cpu ) {
                    $cpuload += $cpu->LoadPercentage;
                    $i++;
                }
            }

             $cpuload = round($cpuload / $i, 2);
             return "$cpuload%";

        } else {

            return false;
        }
    }
}

function translate($text, $fl, $tl){
if (trim($text) == null) return $text; //skip translation if string empty

/* Retain left and right spaces after translation */
$lsto = substr($text, 0, strlen($text) - strlen(ltrim($text)));
$rsto = substr($text, strlen(rtrim($text)), strlen($text) - strlen(rtrim($text)));
/* -- */

/* Declare global */
global $enablecache,$overloadval,$disinterval;
if(floatval(getServerLoad()) >= 1.5)    // Numeral = server load on which to crash
{

    // Write current time to file.
    $fp = fopen("overloadcache.txt","w");
    fwrite($fp, time() . "");
    fclose($fp);


  if($nfh= @fopen("overloadcache.txt","r"))
  {
      $overloadtime = fgets($nfh);
      if($overloadtime > time()-30)   // Numeral = # seconds after overload not to cache!
      {
          $enablecache = false;
      }
      fclose($nfh);
  }
}

/* Check for server overloads  (modification by DeViAnThans3)*/
global $originalEncoding;
if ($originalEncoding != 'utf-8') {
  $text = iconv($originalEncoding, 'utf-8', trim($text));
} else {
  $text = trim($text);
}

/* Check cache for translation */
if ($enablecache) {
  $sql=null;
  $length = strlen($text);
/* 
  if (!mysql_ping()) {
    mysql_close();
    establishConnection();
  }
*/  
  if ($length<=50) {
    $sql = mysql_query("SELECT translated FROM wt_cache_short WHERE originaltext='".addslashes($text)."' AND tl='".addslashes($tl)."'") or die('Query to short cache failed: ' . mysql_error());
  } else if ($length > 255) {
    $sql = mysql_query("SELECT translated FROM wt_cache WHERE originaltext='".addslashes($text)."' AND tl='".addslashes($tl)."' LIMIT 1") or die('Query to normal cache failed: ' . mysql_error());
  } else {
    $sql = mysql_query("SELECT translated FROM wt_cache_medium WHERE originaltext='".addslashes($text)."' AND tl='".addslashes($tl)."'") or die('Query to medium cache failed: ' . mysql_error());
  }
  
	while($t = mysql_fetch_array($sql)) {
		return $lsto.$t['translated'].$rsto;
	}
}
/* -- if not found, proceed with Google Translate */

/* -Establish cURL connection to Google Translate API server- */
$ch = @curl_init();
//@curl_setopt($ch, CURLOPT_URL, "http://ajax.googleapis.com/ajax/services/language/translate?v=1.0&langpair=".urlencode($fl.'|'.$tl)."&q=".urlencode($text));

@curl_setopt($ch, CURLOPT_URL, "http://ajax.googleapis.com/ajax/services/language/translate?v=1.0&langpair=".urlencode($fl.'|'.$tl)."&q=".urlencode($text));

//$postParams = "v=1.0&langpair=".$fl.'|'.$tl."&q=".iconv('iso-8859-2', 'utf-8', $text);
//@curl_setopt($ch, CURLOPT_URL, "http://ajax.googleapis.com/ajax/services/language/translate");
//@curl_setopt ($Curl_Session, CURLOPT_POST, 1);
//@curl_setopt ($Curl_Session, CURLOPT_POSTFIELDS, $postParams);

@curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 (ax)");
@curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
@curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$in = @curl_exec ($ch);
/* -End connection- */

preg_match('/{"translatedText":"(.*?)"}/', $in, $out);
//$ttext = conv(api_strip($out[1]));
$ttext = str_replace ('\u0026', '&' , api_strip($out[1]));
//$ttext = api_strip($out[1]);
//$ttext = unicode_encode(api_strip($out[1]), 'utf-8');

/* Save to cache */
if ($enablecache && (strlen($ttext)>0)) {
  $length = strlen($text);

  if (!mysql_ping()) {
    mysql_close();
    establishConnection();
  }
  
  if ($length<=50) {
    mysql_query("INSERT IGNORE INTO wt_cache_short SET tl='".addslashes($tl)."', originaltext='".addslashes($text)."', translated='".addslashes(trim($ttext))."'") or die('Insert to short cache failed: ' . mysql_error());
  } else if ($length > 255) {
    mysql_query("INSERT INTO wt_cache SET tl='".addslashes($tl)."', originaltext='".addslashes($text)."', translated='".addslashes(trim($ttext))."'") or die('Insert to normal cache failed: ' . mysql_error());
  } else {
    mysql_query("INSERT IGNORE INTO wt_cache_medium SET tl='".addslashes($tl)."', originaltext='".addslashes($text)."', translated='".addslashes(trim($ttext))."'") or die('Insert to medium cache failed: ' . mysql_error());
  }
}
/* -- */

return $lsto.$ttext.$rsto;
}


function callback($buffer){
global $fl;
$modifyhref = false;
//-------------------------------------------------------------------------------
global $enablesession;

// Get language setting from queries:
$lang = stripslashes(@$_GET['hl']);

if ($enablesession) {
        // Save setting to server session [1]
        // If language setting not found in queries, check session data: [2]
        if (checklang($lang)) $_SESSION['lang'] = $lang; else $lang = @$_SESSION['lang'];
}

if (!checklang($lang) || $lang == $fl) { // If invalid language setting or language = original language, skip translation.

$tp = $buffer;
$tp = preg_replace("/<disp_lang>/is", checklang($fl), $tp);
return remove_tags($tp);

} else {
    // Extract BUFFER
$tp = preg_replace("/(^|>\/?)([^><]*)($|\/?<)/e",
             "emodfix('\\1').translate(emodfix('\\2'),'$fl','$lang').emodfix('\\3')",
             $buffer);

if ($modifyhref) {
$tp = preg_replace('/(<a [^><]*href\=")([^"]*)("[^><]*>)/e',
             "emodfix('\\1').langconv('$lang',emodfix('\\2')).emodfix('\\3')",
             $tp);

$tp = preg_replace('/(<form [^><]*action\=")([^"]*)("[^><]*>)/e',
             "emodfix('\\1').langconv('$lang',emodfix('\\2')).emodfix('\\3'))",
             $tp);
}

$tp = preg_replace("/<disp_lang>/is", checklang($lang), $tp);
    $tp = remove_tags($tp);
return $tp;

}
//---end function
}

function langconv($hl, $href){
if ($href && substr($href, 0, 7) != 'http://' && substr($href, 0, 11) != 'javascript:' && strpos($href, "#") === false) {
        if(strpos($href, "?") === false) return $href.'?hl='.urlencode($hl); else return $href.'&hl='.urlencode($hl);
} else { return $href; }
}

function checklang($hl){
$langvar = array ("sq"=>"Albanian","ar"=>"Arabic","bg"=>"Bulgarian","ca"=>"Catalan","zh-CN"=>"Chinese","hr"=>"Croatian","cs"=>"Czech","da"=>"Danish","nl"=>"Dutch","en"=>"English","et"=>"Estonian","tl"=>"Filipino","fi"=>"Finnish","fr"=>"French","gl"=>"Galician","de"=>"German","el"=>"Greek","iw"=>"Hebrew","hi"=>"Hindi","hu"=>"Hungarian","id"=>"Indonesian","it"=>"Italian","ja"=>"Japanese","ko"=>"Korean","lv"=>"Latvian","lt"=>"Lithuanian","mt"=>"Maltese","no"=>"Norwegian","pl"=>"Polish","pt"=>"Portuguese","ro"=>"Romanian","ru"=>"Russian","sr"=>"Serbian","sk"=>"Slovak","sl"=>"Slovenian","es"=>"Spanish","sv"=>"Swedish","zh-TW"=>"Taiwanese","th"=>"Thai","tr"=>"Turkish","uk"=>"Ukrainian","vi"=>"Vietnamese");
foreach ($langvar as $i => $val){ if ($i == $hl) return $val; }
return false;
}

function clearlangsetting(){
global $enablesession;
if ($enablesession) $_SESSION['lang'] = null; /* -Clear lang session data on server- */
}

?>
Hope Dave will include it in next release.

Now - for already installed products
Here we have 2 ways. Easy one, and good one.

In easy one - just drop all tables from DB, set it from new script and change translate.php like described.

In good one - you want to keep already cached data, and this will need some changes.
1. change translate.php like described
2. dissable cache - set $enablecache=false; in translate.php
3. change collate by executing:
Code:
Alter table wt_cache_short collate utf8_bin;
ALTER TABLE wt_cache_short CHANGE tl tl VARCHAR( 10 ) CHARACTER SET utf8 COLLATE utf8_bin NULL DEFAULT NULL;
ALTER TABLE wt_cache_short CHANGE originaltext originaltext VARCHAR( 50 ) CHARACTER SET utf8 COLLATE utf8_bin NULL DEFAULT NULL;
ALTER TABLE wt_cache_short CHANGE translated translated VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_bin NULL DEFAULT NULL;

Alter table wt_cache_medium collate utf8_bin;
ALTER TABLE wt_cache_medium CHANGE tl tl VARCHAR( 10 ) CHARACTER SET utf8 COLLATE utf8_bin NULL DEFAULT NULL;
ALTER TABLE wt_cache_medium CHANGE originaltext originaltext VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_bin NULL DEFAULT NULL;
ALTER TABLE wt_cache_medium CHANGE translated translated VARCHAR( 1000 ) CHARACTER SET utf8 COLLATE utf8_bin NULL DEFAULT NULL;

Alter table wt_cache collate utf8_bin;
ALTER TABLE wt_cache CHANGE tl tl VARCHAR( 10 ) CHARACTER SET utf8 COLLATE utf8_bin NULL DEFAULT NULL;
ALTER TABLE wt_cache CHANGE translated translated TEXT CHARACTER SET utf8 COLLATE utf8_bin NULL DEFAULT NULL;
ALTER TABLE wt_cache CHANGE translated translated VARCHAR( 65000 ) CHARACTER SET utf8 COLLATE utf8_bin NULL DEFAULT NULL;
4. Delete data duplication like described here (note this is very important - otherwise unique indexes will not be set): https://vborg.vbsupport.ru/showpost....&postcount=214

Note that this step if very time consuming and probably you will need to connect to DB with other client that WWW. Remember - always you can use the easy way

5. Optimize tables:
Code:
OPTIMIZE TABLE wt_cache, wt_cache_medium, wt_cache_short;
6. Reindex to unique:
Code:
alter table wt_cache_short drop index originaltext;
create UNIQUE INDEX originaltext on wt_cache_short (originaltext, tl);

alter table wt_cache_medium drop index originaltext;
create UNIQUE INDEX originaltext on wt_cache_medium (originaltext, tl);
7. enable cache - set $enablecache=true; in translate.php


And that is. As I wrote data duplication will disappear for ever from wt_cache_short and wt_cache_medium. Also it will be save in case of hazard described in my 2 previous posts. Also according to my automatic tests - page generation works faster.

So - enjoy
So what happens to saver and cleaner? Do these stay or can i remove them? Thanks.
Reply With Quote
  #259  
Old 07-05-2009, 10:02 PM
Dave Hybrid's Avatar
Dave Hybrid Dave Hybrid is offline
 
Join Date: Mar 2007
Posts: 463
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

We'll I tried again with your new update.

Had to get my host to run this bit as it times out in the browser.

4. Delete data duplication like described here (note this is very important - otherwise unique indexes will not be set):

They say the query is stuck on delete from cleaner...

So i am going to stick with the current update for now.
Reply With Quote
  #260  
Old 07-05-2009, 10:05 PM
NLP-er's Avatar
NLP-er NLP-er is offline
 
Join Date: Aug 2008
Location: Wrocław
Posts: 1,353
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally Posted by Dave Hybrid View Post
So what happens to saver and cleaner? Do these stay or can i remove them? Thanks.
Those can stay for future wt_cache duplication cleaning - this is only table where duplications still can happen. Fortunately there is less records than in other tables, and it is rare to write there translations (till now in my DB i have 380 000 rows in wt_cache_short; 210 000 in wt_cache_medium; and only 65 000 in wt_cache).

wt_cache cannot have unique index for whole oryginaltext and tl, because oryginaltext is too big for that, and making it's unique only for part of it has some disadvantages. I made tests and mysql allows me to set unique index olny for first 323 letters from oryginaltest and whole tl. So if I do that, then data duplication will be a history in this mod - that's the good part. Worst part is that if we will have 2 different translations with same first 323 letters, then only first one will be cached, and second one will be translated by google every time when page will be generated.

I also realize that it is rare to have 2 different posts which starts from such long same text (323 letters), but it is possible - especially when users like to quote others posts. I've just made a query and I have such cases in my cache. So we can have actual solution which allows data duplication, but we know how to delete duplicated data, and once translated text will never be translated again. Or change it to solution where duplication never happens, but we have small chance that some texts will be translated over and over again. Usually when someone quotes other long post, and removes some part at the end – in such case we have 1 long translation of whole original text and one long translation of shorter quoted text. Now both are cashed - if we set unique index on wt_cache then one of those will be translated each time when page is generated.

So - If users want this solution I can give the solution. Who wants it?


But , till wt_cache allows for data duplication - I advice to let cleaner and saver tables stay (you can delete all data from there). Empty tables doesn't bother anyone, and can be helpful for cleaning wt_cache from time to time. In my installation I will keep those at least till I crawl all translation sites with my spider. And probably let them stay after that too.
Reply With Quote
  #261  
Old 07-05-2009, 10:19 PM
NLP-er's Avatar
NLP-er NLP-er is offline
 
Join Date: Aug 2008
Location: Wrocław
Posts: 1,353
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally Posted by Dave Hybrid View Post
We'll I tried again with your new update.

Had to get my host to run this bit as it times out in the browser.

4. Delete data duplication like described here (note this is very important - otherwise unique indexes will not be set):

They say the query is stuck on delete from cleaner...

So i am going to stick with the current update for now.
Just as I wrote:
Note that this step if very time consuming and probably you will need to connect to DB with other client that WWW. Remember - always you can use the easy way

New solution is better and faster. For those who makes fresh install it is same easy as it is right now. For update - 2 ways easy: one and good one.

If you cannot handle the good one, and still want have your cache - so ok, you can stay with old version. But I think that for users it would be good if you test new solution in fresh install (if have difficulties on old one) - see it works fine, and make it official release - for all new users, who I think would like to have faster DB without data duplication.
Reply With Quote
Reply


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 10:23 AM.


Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2024, vBulletin Solutions Inc.
X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.05198 seconds
  • Memory Usage 2,364KB
  • Queries Executed 25 (?)
More Information
Template Usage:
  • (1)SHOWTHREAD
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)ad_showthread_beforeqr
  • (11)bbcode_code
  • (5)bbcode_quote
  • (1)footer
  • (1)forumjump
  • (1)forumrules
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (1)modsystem_post
  • (1)navbar
  • (4)navbar_link
  • (120)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (4)pagenav_pagelink
  • (2)pagenav_pagelinkrel
  • (11)post_thanks_box
  • (11)post_thanks_button
  • (1)post_thanks_javascript
  • (1)post_thanks_navbar_search
  • (11)post_thanks_postbit_info
  • (10)postbit
  • (11)postbit_onlinestatus
  • (11)postbit_wrapper
  • (1)spacer_close
  • (1)spacer_open
  • (1)tagbit_wrapper 

Phrase Groups Available:
  • global
  • inlinemod
  • postbit
  • posting
  • reputationlevel
  • showthread
Included Files:
  • ./showthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/functions_bigthree.php
  • ./includes/class_postbit.php
  • ./includes/class_bbcode.php
  • ./includes/functions_reputation.php
  • ./includes/functions_post_thanks.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • showthread_start
  • showthread_getinfo
  • forumjump
  • showthread_post_start
  • showthread_query_postids
  • showthread_query
  • bbcode_fetch_tags
  • bbcode_create
  • showthread_postbit_create
  • postbit_factory
  • postbit_display_start
  • post_thanks_function_post_thanks_off_start
  • post_thanks_function_post_thanks_off_end
  • post_thanks_function_fetch_thanks_start
  • post_thanks_function_fetch_thanks_end
  • post_thanks_function_thanked_already_start
  • post_thanks_function_thanked_already_end
  • fetch_musername
  • postbit_imicons
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • postbit_display_complete
  • post_thanks_function_can_thank_this_post_start
  • pagenav_page
  • pagenav_complete
  • tag_fetchbit_complete
  • forumrules
  • navbits
  • navbits_complete
  • showthread_complete