First time (sort of)..?

I just had sex with my partner for the first time a few hours ago .. It felt good, but not as amazing as I expected, and it really really hurt so I couldn't go for very long at all (I have horrible pain tolerance). There wasn't any blood or anything though. The thing is, it's not really my 'first time' because I was raped a few years ago ... there was no blood then either, but I don't remember the actual sex part hurting. My partner is a lot bigger though, if that has anything to do it.
I'm worried that it's always going to hurt and so I won't be able to have sex for very long because of it. Is it meant to hurt even after the first time/why does it hurt? How do I stop that? Is there something that's meant to 'pop' and then it'll feel better for me?