Mixed emotions with a sex problem

Em
I am in some need of some advise. Ever since me and my SO have started having sex three years ago the sex has been great. Expect for those times when my SO cannot stay hard.. It used to happen every once in awhile and of course he was always embaraased and I always felt like it was me. But it has happened the last two times we have gone to have sex and it has really dawned on myself that I feel like it's me. I feel as if he maybe isn't attracted to me anymore? I'm not sure.. And of course he won't tell anyone about it. So, I was just wondering if anyone else has ever had similar experiences with their SO?