Scared of People Touching My Belly...Advice?

My first pregnancy was at the end of my senior year. I was with my now husband but we were long distance. An ex I went to school with found out I was pregnant. He proceeded to come to my home when I went home sick one day to "check up on me". I wouldn't let him in my doorway so he began pushing me around and I ended up falling....it caused a miscarriage. Now I am 6 months with a baby girl. Everytime a stranger goes to touch my belly I nearly cry and cringe. I don't even have the courage to tell them no. How can I get over this fear? Does anybody think it is irrational?