(Sorry, i know its an old thread.)
I'm still running vB 3.7.4.
I changed the size of the popupwindow in the "editor_smiliebox" template.
and added aditional smiley rows exactly like ReCom discribed.
It works well apart from one thing:
Not all elements of the background in the popupwindow adjust to the new window size.
Screenshot:
I searched in the "editor_smiliebox" and "smileypopup" templates and in the "vBulletin Options" in the admin,
but could not find were to edit the width of the background table inside the popupwindow.
Can anyone help please?