There isn't an option to do that (why? I don't know.). If you just want to remove it from the navbar, then you may put a condition around it in the navbar template:
HTML Code:
<if condition="is_member_of($bbuserinfo,x,y,z)"><td class="vbmenu_control"><a href="$vboptions[bburl]/memberlist.php$session[sessionurl_q]">$vbphrase[members_list]</a></td></if>
Change x,y,z to the usergroup ids for the groups you don't want to see that link. I think the link is in a couple of places in the navbar.