What version vb are you using? I'm testing this on my 3.8 board and my moderator test user cannot see the Invite Only groups that he is not a member of. The test user gets a No Permission screen when trying to view that group.
Try disabling your modifications/plugins and see if you still have this problem.
Note: To temporarily disable the plugin system, edit config.php and add this line right under <?php
PHP Code:
define('DISABLE_HOOKS', true);