Dick going limp in the middle of sex??

My boyfriend and I just had sex a few hours ago, we were making out he fingered me, I was rubbing his dick, he put the condom on, starts fucking me and then loses it! He goes completely limp. This has happened a few times and I can't even deal with it sometimes and it completely kills the vibe. He took off the condom and begs me to keep making out with him and I don't even want to do it anymore. I felt bad so I kept making out with him and he got another condom starts fucking me again and then starts getting limp again. How!!?? Is this normal??? Am I doing something wrong? I already feel weird about our relationship and have contemplated breaking up with him. I just feel like there's no more chemistry. He lost his boner twice, and after the second time he just stayed inside of me trying to get hard again and he finally got hard and kept fucking me but it wasn't even enjoyable? Sex right now just feels like such a hassle and I feel like that's when I should maybe call it quits? It's been like this for months now and I don't know if I can do it anymore. What should I do?? Is going limp in the middle of sex normal sometimes? It's probably happened 10/50 times we've done it. That's not good right?