[tbworld]

That is a bit more difficult. One is a plugin for the Ckeditor the other is not. I have a feeling vbulletin had a problem implementing the smiliebox code in the container div -- what you are referring to as the "more" popup box -- as an internal plugin.

I will mess around with it a bit and see what I can see.

--------------- Added 27 Sep 2013 at 18:02 ---------------

I just worked this up for another user on vbulletin. Simple hack but it might help you for now, until I look at your proposal -- which is interesting.

Hack: Set the minimum height and width of the Ckeditor plugin smiliebox

Edit:
Use a good editor (Do not use Notepad or Wordpad or something similar.
You can use the vbulletin uncompressed source or just edit the compressed since it is just a few parameter
changes.

Edit in vbulletin file area:
"/clientscript/ckeplugins/vbsmiley/dialogs/smiley.js"

Search for "270," <-- Do not include the quotes

The code you are searching for is the following.

        minWidth  : 270,
        minHeight : 120,

Change this to the minimum width of your desired popup box.
NOTE: This should be less than the smallest device resolution you support.

Examples:
minWidth : 450,
minHeight : 290,

minWidth : 610,
minHeight : 450,

It is your environment so experiment.

Save and Recompress:
For final production version always recompress.
YUI compressor is easy to use and does a good job most of the time.