Sex time

Help!! When me and my SO have sex it's not long st all, he finishes almost immediately... like literally like a minute or two in and it drives me nuts bc I'm ready to go but he finishes so quick.. he has told me that he lasted for forever with his past relationships but with me he just can't. Is that a bad thing? Am I not doing my part?