Fiancé can't stay hard!

So my fiancé is having a difficult time staying hard during sex. It's starting to frustrate me but idk what to do. I ask him what's wrong and all he says is that he doesn't know ?. He says it has nothing to do with me but I feel like it does sometimes, I feel that's one reason why we're having a hard time TTC. Is anyone else having/or had this problem with their man? Any suggestions??