Should I allow this to make me feel insecure?

I recently met a guy and we’ve had sex twice already in which I felt his penis was too small the first time and I decided to give it another chance. Well his penis is still small compared to average. Anyway, he takes so long to cum.. I usually never have that problem, most of the time sex doesn’t last this long with other men. He takes forever and it makes me feel like I’m not pleasing him enough or I simply just suck (which I honestly don’t think I’m that bad). I legit was like “damn you take forever to cum” lol but anyway.. is it me or could it be him?