Boyfriend is 23 and can't keep a boner? Help?

My boyfriend is 23 and we have been dating for a while now. When we first met, he was 22 and he was a virgin. Almost every time we try to have sex, he loses it or either he comes so fast. Most of the time he will lose it as soon as we start. Like I said, we have been dating for a while now and are very comfortable with each other, so I don't think it's a nervous thing. I don't think it's an attraction thing either. He seems very attracted to me and loves doing other things to me. He also is really fit and works out pretty frequently, so I wouldn't think it would be anything like that either. I've had lots of experience in the past, so this is unusual to me and sometimes hard to deal with. Any tips or solutions would be greatly appreciated!