AdminCP > Styles & Templates > Style Manager > Edit Templates > Private Message Templates > Double click pm_messagelistbit.
Find:
<a href="private.php?$session[sessionurl]do=showpm&pmid=$pm[pmid]">
<if condition="$show['unread']"><strong>$pm[title]</strong><else />$pm[title]</if>
</a>
Change to:
<a href="private.php?$session[sessionurl]do=showpm&pmid=$pm[pmid]" target="_blank">
<if condition="$show['unread']"><strong>$pm[title]</strong><else />$pm[title]</if>
</a>