As Mindtrix already posted, you can specify which forums have their subforums ordered in columns.
on that screenshot i entered to show the subforums of my 2 categories to show up in that specific columns, but you can change which one's subforums show up that way.
But note, this just works on Forumhome, on forumdisplay it will show all forums the traditional way.
also i have not tested about how nested forums will look like if you have enabled to display more than two levels on forumhome, maybe it would be worth a try someday