So the cause of the line breaks is part of the VB BB code parser. When you disable HTML, it sets the flag:

$donl2br = true;

This in turn causes the do_parse method to replace newlines with <br>'s.

I may be wrong, but it looks like the only way to turn off the br's is to turn HTML support ON.

I found another hacky way around this system that only works on my article system where I directly call the do_parse method... but this is not a general solution.