He won't stay hard .

We've only been dating a couple weeks but we've known each other for a long time . Ever since the first time we had sex we would start ... everything would be just fine till a few pumps in he just stops and his dick gets soft . Giving him head it's fine and he's giving me head it's fine but he sticks it in and it's over. I tried talking to him about it cause it bothers me , starting to think something is wrong with me or crazy Shit like (is my pussy too big/loose ?) (Do I have a smell) (am I sexy enough ). ... he swears it's cause he's nervous there are other people in my house but it still happened when no-one was home ... I tried to break up with him cause I couldn't take it anymore its making me sad .. but he doesn't wanna give up that easy . He wants to get away from the house and go to a hotel ... have a nice romantic night and have good sex lol . I'm down except I'm so nervous ! What if it happens again . What do I do what do I say ..... IDK