While pumping at work, do you have coworkers complaining that you're in the break room pumping?

I don't pump in the break room during lunch time (12-3), but there's always someone outside the door whining or complaining that I'm in the room. Multiple times, I've stopped in the middle of pumping to let them get what they need, but it's just annoying to me.