abuse?

So I dated a guy about a year ago. He was my first boyfriend and pressured me into oral sex and lots of things that I was uncomfortable with but I went along with it because I was naïve and young. He owned a maroon van with the back seats taken out so if we ever got down we would do it back there. One night we were making out, I was naked apart from my panties, which was fine. But he started to finger me. I had told him many times that I didn't enjoy that and I didn't like it when he did it and he always backed off when I told him to stop. This time I kinda pushed his hand away and said, "maybe we should cool off," and started to shift away. He grabbed my legs and pulled me back toward him, continuing to finger me, then moving my underwear and getting more intense about it. I started trying to push his hand away but he grabbed my arms and pinned them down. I'm a very petite person (about 4'11) and he was a big guy. I started to cry, and kept trying to get away because it was very painful and I was very scared. After a bit he let go of me, told me to calm down and drove me home. We broke up awhile later and as I mentioned previously, it's now almost been a year. My current boyfriend and I have begun getting intimate and there have been a few times when he has tired to finger me and I shut down and make us stop. He is very kind and compassionate about it. The other day we were laying down kissing and he started to finger me and my eyes were closed and for a second I thought I was in the back of that van again. It was like I completely forgot where I was. I had a huge anxiety attack and started to panic. He was very sweet and held me the whole time. Is there something wrong with me? I know my boyfriend will keep me safe and won't hurt me so why am I still worried about it? Will I always be afraid of being fingered? Was what my ex did abuse? Or was it just what happens?