Could this be affecting our ability to conceive?

TMI, but my husband and I have been married three years. When we first got married, he had no trouble maintaining an erection. Now he never seems to be hard enough and finishes very quickly, like a minute or less. Up until a couple of months ago, he had a very stressful job situation, and we thought the stress was affecting his ability to stay hard. But now there’s no such stress and it’s still an issue. Could that be affecting our ability to conceive? Also, I have asked him repeatedly to consult a doctor about this, and he has not.