Do you tend to work in the back? If the members don't see you they assume you're not doing anything.
Thing is the stuff on the back (managing the staff, fixing code, adding things to the site, etc etc) tend to go un-noticed. But they have to be done, and most of the time I'm happy they don't know I'm the one dealing with it anyway.
At any rate this is just a popularity contest, and they always hurt someone in the end. I wouldn't worry about it..