Please Tell Me I'm Not Wrong

My husband and I have been TTC for two years... 
Nothing medically is wrong with me from what doctors have said. I have an arcuate uterus, and Elders Danlos Syndrome. Doctors have said that I should be good to get pregnant.
We have looked at all options for me.. Not him. He has been depressed and stressed and I feel like it could be a valid reason for us not conceiving.
I mention this and he basically called me stupid forcing our lack of success TTC on me. 
I'm mad, hurt... Am I wrong?