a baby daddy story

So the sperm donor for my daughter is scum. Literally scum. In prison for life. I left him during the pregnancy and ended up with my now husband. A wonderful guy, Jason, and I love him to bits.
We just did a TPR (termination of parental rights) and it got pretty messy in the middle. Sperm donor would write me all the time, sending hurtful things. One of them said that I would never have a baby with Jason---that HE would be the only one that would ever love me and his baby would be all I would ever get.
The TPR passed, thankfully. 
But here I am, having issues getting pregnant. It's not my husband--we've both been tested, and it's me. Even though it's only PCOS and tons of women get it, I can't help but be haunted by the words of SD. I'm afraid I waited too long--too log to start the TOR---too long to try to conceive---and it breaks my heart and feels me with fear. 
Anyway---I just needed to get that off my chest. I've been crying a lot tonight---I haven't been taking my depression meds since TTC and it's so easy to take it out on others if I don't get things off my chest. 
Thanks for listening.