Clomid fears after miscarriage

Background: I was that person everyone hates who got pregnant without trying the first time around. My daughter was a surprise, at exactly the wrong time, but we love her to bits and we've been wanting another for quite some time! My doctor put me on clomid back in December after finding out I was no longer ovulating on my own. It worked and we got pregnant with our second little bean! Unfortunately, we miscarried at 5 weeks, and I did extensive research on clomid and found out it raises your chances of miscarriage and birth defects. Since then, I have been too scared to use it again. We continued trying using natural methods (myo-inositol, maca root, royal jelly, ect). I went back to the doctor and found out I am still not ovulating, so after a tough debate, we've decided to give it another go, but I'm terrified. I took my first pill yesterday and now I'm panicking, thinking 'what have I done?' I want another baby SO badly, and it breaks my heart how much my daughter wants a sister, but I'm scared of the heartbreak of another miscarriage, or having a baby born with severe defects because of the drugs. I feel like if I did get pregnant with the use of clomid, I'd be waiting for the painful diagnosis of what it caused and I wouldn't enjoy the pregnancy at all. I would be holding my breath until the child was 2-3 years old, and I could see for myself that they were healthy and unaffected. If anything was wrong, I would blame myself for it. How do I get over my fears? Should we just give up and say it wasn't meant to be and enjoy our beautiful little girl? Or do we keep pushing to complete our family even though it terrifies me? This is just a rant to get my feelings out there, but if anyone has any advice, words of kindness or support, or anything else to add, please feel free. I'm just feeling so overwhelmed and like I don't know if I'm making the right choice.