Soft during sex

I’ve been with my boyfriend for 5 years and and I notice that sometimes halfway through sex he randomly will go soft and say it’s because it feels “too sensitive”. This has def happened here and there since the beginning of our relationship and tbh I kind of thought it was normal but none of my friends can relate with their partners. He’s not on any medications (he does smoke weed though) and is not a heavy drinker. We don’t use condoms and I don’t think it’d be performance anxiety as it’s not a new relationship. With all that considered my ego feel a little bruised and wonder if part of it is me even though he swears it’s not. Anyone else been in this situation?