He can't stay hard

My husband and I have been together for almost 5 years now. We are both turning 20 this year and got married last October. Well for the past year or so, there have been times he can't stay hard during sex. Either I'm sucking on his private area or riding on him, it usually goes limp. He reassured me that it wasn't me and that it was him. Of course it made me a little imitated because I really thought it was me. But he's not to sure as to why this is. I doubt age is the reason. We're both still 19. We had a lot of sex. He gets a boner pretty quick but like I said, half way through he goes limp. Have any of you ladies experienced this with your partners? I have mentioned to see a doctor but he isn't to keen on it yet.