Remove them from being displayed where? If you don't allow users to have an avatar, then a default avatar will be shown. I suppose you could replace that default avatar with the clear.gif image (a 1px x 1px clear gif). Otherwise, you would need to edit all the templates to remove the code.
leaves a little box, thats a bummer. Just kinda don't like a bunch of blank avatars showing until more people log in with facebook, etc... Thanks for the idea though!