he can’t keep his hard-on

my boyfriend and i are our firsts when it comes to sex, and i’m his first when it comes to everything sexual. however, during sex he sometimes loses his erection and then gets annoyed at himself. i ask him if it was because i turned him off, but he said “believe me, you aren’t turning me off at all.” so what could it be if it’s not me? we’ve had occasional sex since november. sometimes he cums quickly and apologizes to me.