Am I allowed to be upset? (rude conversation I had, mention of miscarriage, potentially upsetting)

So I'll start by saying I've never had a miscarriage, my fiancé has with a ex, him and I have been trying to have our first for what feels like a long time to us. But what happened while we were out with two of his friends (on a double date) upset and kind of pissed me off. So his friend was driving and the four of us had been talking about pregnancy because both the other girl and I want children soon. But it's when the other guy said "I've got strong swimmers but I never have to have a baby cuz they all die" of course I made a surprised scoffing noise because it shocked me, his reaction to the noise I made was "oh and it doesn't matter if you've had a miscarriage I lost a baby when my ex was five months into her pregnancy." He doesn't even really know me I don't understand how he felt it ok to disregard any potential loss of someone else. And my fiancé didn't tell him to shut up or anything. I was upset just by how mean and hurtful what he said was. I ended up drinking (which I never drink) just to try to ignore what he said. But I now feel like I had no right to be upset since I haven't lost a baby, but struggling trying to conceive over the last year has been upsetting in its own way, I would do anything to have a baby and here is think guy joking about how he always gets girls pregnant but they always miscarry. Ugh.