Am I wrong for letting this bother me?

So recently I've been bothered by a particular event in my husbands past that I didn't know about for a very long while into the relationship. About 6 years ago when he was with one of his ex's and in the 2-3 years they were together, she had 3 miscarriages. He never thought to tell me this, even when we began to discuss having a baby together. I suspected something had happened in that relationship, so I harped on it one day and he told me unwillingly. He said I didn't need to know cause it doesn't matter because they were never born. It's bothered me ever since, but even more so lately knowing that I might possibly be pregnant. Am I wrong for letting this bother me?