Anyone scared of history repeating itself?

I had 3 kids with my ex husband. He was really abusive and left me and the kids when the twins were 2 and my oldest was 4. He ran off with a coworker. I'm remarried and my husband now adopted my kids and is desperately wanting a baby. He's got no biological kids so this is something he wants so bad. I love him and I know he's good but everything that happened terrifies me. How do I explain this to him? I told him I'm worried but he's hurt and doesn't understand.