Hmm, works just fine for me as you described it.
http://kirbydemos.ath.cx/forum
Nobody can see Category A (
http://kirbydemos.ath.cx/forum/forumdisplay.php?f=9), but Users in Usergroup X can access Forum (
http://kirbydemos.ath.cx/forum/forumdisplay.php?f=12) and Users in Usergroup Y can access Forum Y (
http://kirbydemos.ath.cx/forum/forumdisplay.php?f=13).
Testaccounts
testuserx / test
testusery / test
But I think this is not what you want

You want to give everybody access to Category A, right?