How do I deal when the guy gets soft from wearing a condom?

Help! So I’ve been seeing this guy and we’ve been sleeping together. I’m not on the pill yet (I’m working on getting on it so save the lecture), but he has a hard time having sex with condoms, he goes limp pretty quickly. This leads to my dumbass having sex without a condom and relying on withdrawal. My weak self has done this twice, including tonight (4 days til ovulation but I took plan b last week, which hopefully delays it). What can we do so that he stops going limp from wearing a condom?