Is "quickly sex" a problem?

Our sex is good but it's too quickly, and I always end up wanting more :(
I'll try to keep this breef: Me and my boyfriend were in a relationship for 2.5 years then we broke up. But after a time, we got back together (that was one year ago). In total, we have 3,5years together.
In the beginning, sex was good but I wouldn't have a orgasm, but since we got back together, 95% of the time I have one orgasm.  
Which is wonderful!!! BUT: our sex is very fast and only once a week!!!(sometimes we spend 2weeks without it)
And he doesn't ejaculate every time!!
Is that a problem?? He should ejaculate?
For me, we would do a lot more sex during the week and it would last more than a hour, but now we only do for 15/20 minutes or something :(
Another thing: it looks like I'm the only one who wants to have more sex everytime. He said he is ok just laying beside in bed a hugging each other.
 Am I doing something wrong? Is that normal?
He was the only one that I had sex in my life, so I don't know if that's normal or if he isn't interested anymore :(