Sex Pain

So me and my boyfriend have only had sex a few times. Half the time, its amazing with no problem. The other half, it hurts when he goes deep inside me. Is it because I'm not turned on enough yet? I do worry about it hurting so I know that doesn't help, but should i be concerned about this?

(Side note, I was a virgin before we started dating so the few times we've had sex is my only experience)