sex drive

My boyfriend complains that I never want to have sex with him. I told him it's because I just don't have a high sex drive because I had been single for 7 years before we started dating 
But is it possible that I don't want to have sex because I don't find him physically attractive and so far all the times we have had sex it was almost never satisfying for me since he would be done and out in less than 5 minutes?
I feel bad that I never want to do it and he always wants to and tells me that I don't love him