Condom slipping...

My bf and I had sex for the first time, i enjoyed it but i noticed that the condom would slip off sometimes...does that mean he wasn't enjoying it? (He couldn't stay hard) Or the condom wasnt the right size?