Custom Profile Fields and Performance

[squishi]

For an extension that I am creating, I have added many custom profile fields.
The vb_userfield now has 71 fields. About half of those fields belong to my extension.

These extra fields are only necessary in my extension. I guess they are loaded in other parts of the forum as well, where they are actually not needed.
I could remove the extra fields and add them to a special database. The downside is that I would have to do all the validation in my script.

From a performance stand point, should I remove the custom profile fields?
Should one keep the field count as low as possible?