realizing my first time was non consensual, years later

We started dating my freshman year of college. I had been with this guy for a few months, he was my first real boyfriend. I grew up very conservative sexually and was saving myself for marriage. He was always very pushy about initiating things that were physically beyond my comfort but not violent, he would just beg and then would do something anyway even if I seemed hesitant. He tried to get me to have sex with him almost every day, but I made it VERY clear over and over and over  that I wasn't ready and didn't want to have sex any time soon. That's part of why I would at least let him finger me, etc, because I hoped that would be enough. 
  One day he, I thought, jokingly said that we were gonna have sex today and said "lol yeah right"..  I was at his house later we were messing around , things started getting way too close for my comfort, he was pushing his penis against me near my vagina.  "hey, what are you trying to do" he said "don't worry, I swear I'm not gonna do anything, I just want to be near you." I made it clear that he was not to go any farther. but the next thing I knew he put it in me anyway. I was so shocked I didn't know what do do, so I didn't do or say anything, I just let him. I don't even remember the rest of it, except for that I drove home and spent the day sobbing  and feeling awful.
I convinced myself it was my bad for letting things go as far as they were, that it was bound to happen and pushed it aside. we were together for almost 3 years afterwards, and I did love him.  Even though he  was manipulative and gaslighting the entire time. Not to mention that he would have sex with me while I was asleep, blow past my boundaries or beg me donto things. Part of why I stayed so long is that I believed I was only supposed to have sex with one person (re:religion) and so I needed to make things work.  
When I finally broke up with him, he stalked me on and off for a couple of years. I don't think he'd ever consider what he did as wrong and I'm worried he just feels like that's how to be. 
Does this classify as rape? Its been 6 or 7 years now and I'm in an amazing relationship now. I haven't even thought him at all until recently and suddenly I'm realizing how fucked it that actually was and I am thinking about it all the time, trying to figure out how to navigate this.