On my local dev site, I set up 3 child forums, and for the restricted usergroup, set all permissions to "No" with the exception of "Can View Forum" and then logged in under a restricted account, and this is what I see:
It appears that you likely have "Can View Others' Threads" set to "Yes."
In my opinion though, if a user is not allowed access to a forum, then they shouldn't even see that the forum exists, and I would also set "Can View Forum" to "No" (I would set all permissions to "No").