Sex problem

My boyfriend and I had our first time recently. And, he kept slipping out of me. I really doubt it's his size, he's about average. But I don't know what it is, and it kept ruining our rhythm each time and we had to stop to put it back in. Keep mind, we used a condom, so could that have to do with it? Or is it just the way our rhythm is, since it was our first time?