Soft During sex

Sex with my boyfriend is great but there’s one problem. Every time he takes his dick out of my vagina for more than 5 seconds or so he goes soft. I have no idea why. It’s not like I’m not turning him on and when I give him bjs it’s fine. Even when we’re cuddling he’ll have a boner most of the time when we’re not even doing anything other than making out or whatever. I asked him if I was just bad at sex and he says no, which I believe bc when he’s inside of me it’s great, but it gets annoying to have to wait for him to get hard or keep changing the condom. What should I do? Is it my fault?