People rubbing your belly...

Do you ladies ever have people just come up and rub your belly? I have a few co workers who never ask. It makes me feel so awkward because they're men. They just say they're so fascinated with the baby but I don't know how to nicely ask them to stop. It's been annoying me lately because I feel like it's getting inappropriate, what do I do? If I said anything to my manager they would know the message would be directed to them, they're the only ones who do it. It's getting so bad that I when I walk through the office, I go out of my way to avoid their stations.