natural selection

Megan
Quick background... DH and I have been TTC for ~4 years now. I am overweight, don't have PCOS, had surgery this past Dec. and had one cyst removed, some adhesions between one ovary and my uterus. and a very small spot of endometriosis. My tubes were open according to my doctor. Ovulation tests and regular periods indicate that I ovulate every month... general consensus from my ObGyn is that I'm prolly not the problem. DH had a semen analysis ~2 yrs ago and results were low density, low motility, and low count. Went to my ObGyn and he has referred us to a reproductive endocrinologist. He thinks that doctor will tell us we need to try <a href="https://glowing.com/glow-fertility-program">IUI</a> because of DH's results. 
So, as DH and I are discussing all of this last night he posed an interesting question. If he has such "bad swimmers" and we try <a href="https://glowing.com/glow-fertility-program">IUI</a> or whatever means to try and get pregnant, are we at a higher risk for birth defects? Is this God's way of saying you are infertile and aren't supposed to have children? I couldn't find anything that said if you are infertile and undergo fertility treatments that you can experience higher rates of birth defects... has anyone heard anything like that before? Or is DH paranoid?