Why do I always struggle for ages to solve something - then give up and post the question on a forum.
And 2 minutes later I figure it out?!
I found it in Admin CP -> Options -> Message Posting Interface Options -> Smiliebox Total Smilies
And I made two groups of smilies, with a different display order for all the non standard ones, and limted the show smilies to 11 - so now it's fixed