Feeling emotional

I work at a residential building and for the most part residents have been very thoughtful and full of blessings towards me and my little girl; I'm 38 years old I already have a 20 year old daughter and expecting another girl any day now. One older man asked me when I got married I told him I wasn't; "he then said so you were fooling around irresponsibility" I was in shock and felt ashamed for 2 mins then told him I was very happy and blessed with my family. I wanted to get out of my chair and slap him for saying that. But it doesn't matter what other people think or feel about your life all that matters is how you feel.