Boner goes down...

I’ve been with my boyfriend for 3 years, in the last month every time we go to have sex his boner goes down.

It happened once and ever since it’s been an issue, when I asked him about it he says he worries it’s going to happen again and that might be why.

Is there anything I or he can do? Or anyone else that had this problem can shed some light? Thank you😢