my boyfriend can't stay hard.

There have been a few situations where my boyfriend can't seem to stay hard. We have amazing sex most days but there's always a chance that his man parts are cooperating. Wearing condoms immediately makes his dick go from rock hard to soft. I'm not sure if it's cutting off the circulation but we've tried several different types of condoms. He's only 20 and I know this isn't common but I wanted to see if others have this problem before going to the Doctor. He says he's sexually attracted to me and I have no doubts, we have the best sex ever when able but this happens once a week or so and he gets so frustrated.