Sex, regret, sexual abuse? thoughts? (long)

Panda
Re-named because I realized how I describe things may be triggering to some.
Hi ladies, I came on here because a friend told me about it, and I just want to...I guess talk through my thoughts with more people in a less anxiety-inducing way. I was in a relationship that was at many times manipulative and controlling, and I certainly had my faults, and do, but I've had something I wanted to talk to someone about for years now.
I dated my ex for about six years, and broke it off with him a few years ago.
I've dated a bit here and there but not much and nothing serious since then.
I just can't get to where I feel comfortable for a few reasons.
The biggest being, when we started dating, this guy was very "pushy", and I was very young. 
I had only ever kissed one other person before that and never had sex.
From saying "I love you" to allowing things like groping, "griding", manual sex, head, to sex, I would express discomfort at each "level". I'd say no, or I wasn't ready, and he'd say ok. Initially, he would back off, then go for it again, or just do what he wanted.
It got to the point he didn't stop unless I cried or actually very forcefully pushed him away.
A couple of scary times that didn't work, but whatever he was going to do, he stopped when my sister came home right at that moment. And he laughed about it.
He guilted me for being hesitant, for "just laying there", for feeling guilt and shame (I am from a religious family so I believe there's an element from that plays into it).
But in all that time, I felt more like it was all something I allowed to happen, vs. something he did to me, you know?
And, I can think of, I guess three times where I actually initiated things in that almost six years.
The other times, he was pushy, and sometimes things were maybe more consensual for others (but again, I had hesitation. Some because of religious feelings and I see now a lot because I just didn't want to have sex with him).
But I felt abnormal for feeling that way, pressure to make him happy, and very confusingly, my body responded in ways contrary to what I was thinking and feeling very often.
So I don't know how to think of that experience, but it's been nearly four years and I've been with no one since and it scares me. I want to get over/through this. I'm nearly 30, I do have trust issues but I am in counseling and I want to be able to have a relationship again some day.
But I don't know what to call what he did or explain what it was that was so upsetting. Because I participated, to an extent. Sometimes I didn't say no, but I don't know why, I tensed up, and just "got through it", and was confused, because like I said, my body felt "good", but I didn't. 
Am I making sense? I feel like a freak.
So I guess my actual question was, did I feel guilt only because of religion, and let that distort I "really wanted" (I've been told by one person that o felt bad because I sinned, and wanted to shift blame, vs. something wrong being done), and/or because I have negative feelings about him the way I perceive the past is colored, or is this common?