Please help..

I’ve been seeing this guy for maybe about 3 months, he’s sweet, nice, makes little dumb mistakes, spoils me, etc. We both work mon-fri so we see each other every weekend , he lives about an hour away & he comes down to visit me we go to church spend time with fam etc. We tried having sex & his dick won’t get hard/stay hard.. I have a very high sex drive he went to the doctor said nothing was wrong with him, he says he loves me it has nothing to with me. During the day he works comes home to sleep & I work overnight shift & day shift we barely get to speak & when we do is about arguing how he says he’s a bad bf & how we barely speak.. should I just end it or keep holding on?