Stressed beyond anything ever!

Me an my husband have been together for 7 years married for 3.. I've wanted a baby for a long time but we weren't financially set so.we waited. And he has a kid with hos ex an she's always trying to mess with our relationship an I know he wants nothing to do with her but it's still always there in the back of my mind that I am never going to be the only one he's had a child with.. so I just feel like it's not as special anymore. I know that's probably crazy but it's just eating away at me. And I want a baby more then anything, but I know when I do get pregnant she's just gonna try even harder to get between us.. and I'm afraid the stress of a baby might drive him away from me (not into her arms) but enough to the point where she'll never leave us alone. I know it probably makes me horrible to say I wish he didn't have a kid with her an I really wish I didn't feel that way but it's just I'll always be second fiddle in everything baby related.. 😩 it's just really stressing me out an I needed to vent.