Family members suggesting I don't have another child?

I am genuinely curious if other moms get the same type of questions or it's just me???

Backstory:

My fiancé and I have been together almost 4 years. He is Dominican, I am not. We have a 2 year old boy. We live a comfortable life. Earlier this year an older female cousin in my fiancé's family was playing with our son and asked me, "This will be your only one, right?" Meaning, my son will be my only child. I was a little shocked when I answered, "No, we're having more.. just not right now." The woman raised her eyebrows and kind of smiled, but not in a warm way. Later that day I asked my fiancé if he felt weird about the comment and he brushed it off, didn't bother him one bit. So I tried to forget about it, even though I tend to over think things.

Fast forward to today. We met a different aunt in the family while shopping (from a different side of the family). This woman was older also. Within 5 minutes of meeting my son and I for the first time, she looked at me and said, "No more babies for you, right?" While pointing at my torso area. I said "No, we want more.." the woman shook her head and said, "It's hard..." commenting that dealing with children is difficult. This time I was offended. ESPECIALLY because my fiancé and I have been actively trying to conceive and no one knows it yet.

I would love to not feel the burden of other opinions, but it makes me question... how will people respond once we are pregnant again? Should I not care...even if it's his family? Is this a cultural thing, that people feel the need to give advice and opinions on whether or not we should expand our family?

I don't understand it but it's bothering me. I can't shake it.