boyfriend can't stay hard with a condom and it's affecting our relationship

I started officially dating my boyfriend relatively recently. He's absolutely great, like the kind of man I've always dreamt of. We connect insanely on every level except one: sexually.

We're both relatively young (early 20s), and somehow I am one of the first partners he's ever used a condom with. He says he hasn't used them in years, but that he doesn't want me to go on birth control for him (which I won't be doing for personal reasons). Anyway, after multiple attempts, it has become evident that he normally can't stay hard with condoms on. The past three times he has went soft after a minute or two of sex, to which he pulls out and gets angry (at himself, not me). He has only came from sex once, and that was using an extra thin condom. Even with the extra thin, he still goes soft normally. I've tried to be as comforting as possible but I know he feels horrible about it and he expressed that he feels like he's jeopardizing losing me if he can't figure this out. I reassured him that that's not true and I have no problem working through this with him, but I know it's still bothering him in the back of his head. I've definitely never seen him get that angry before. He swears that it's nothing to do with me and that he's somehow just not functioning right. I feel bad because I try to reassure him as much as I can but he evidently still feels dysfunctional. In a burst of anger he said something along the lines of this not working out if we can't figure our sex out and it was sad to hear. We're both insanely attracted to each other so I don't know where we're going wrong.

I'd really appreciate any tips or insight for this situation. My boyfriend is so great and I could really see things working out with him long-term, but it'd be great to be able to figure this issue out (more so for him than me, sex is really important to him).