I don't know if I was sexually abused

Hi! I don't really know how to start..
I'm 23 and recently I've been deeply strugling with my past. When I was a child, not sure how old I was at all - I was young enough not to know how old I was. Maybe 4 or 5? No idea but very young. I have always loved receiving massages from people since 8 was very young - like foot massages or having someone play with my hair for example. My grandfather - on my father's side would be one of the people that liked doing this. One time he put me on his lap, he was sitting in those reclining chairs so he was almost laying down. I was laying on top of him. He started touching me and "masaging" me in a regular and non sexual manner but then he would start putting  his hands under my clothes. And then under my pants.. And under my underwear... He touched and played with my vagina. This happened a few times. I always closed my eyes and pretended I was asleep. I was young but I knew it was wrong and I felt ashamed that I knew that was wrong so I just pretended I was sleeping. Everytime someone would walk into the room, he would stop. I knew it was wrong or he would not stop. He only touched the outside of my vagina, nothing ever went inside of me. This has always bothered me and it's my deepest secret. I've always know it was wrong and I always shrugged it off. Over the years, when I layed on the couch at their house, he would always massage my back or something and he would always reach too deep under my clothes. I always had to lock my arms wround my boobs so they wouldn't be touched. He hands would go down my shirt in the front and in the back. It makes me sick that I let him do this for so long, even after I was so old. This past holiday season this started to really bother me. I can't stop thinking about this. My grandfather touched my ass a few times last december but was it on purpose? Or did he just brush his hand on my ass by accident bc he's an old man? Does he not see I have boobs now and he shouldn't pat my chest as if it were my back? Is this on purpose? Or does he just see me a child in a non sexual way? Why do I have to cover up and hide my body for "family"? I don't know any of these questions. One day I broke down and cried for hours on the floor - I couldn't move. I promised myself I couldn't leave my room again, for the rest of my life, if I didn't sum up the courage to tell my mom and make this whole situation real. I had never told anyone or thought about this situation in a more concrete manner. It was always something weird that happened to me once when I was a little girl. I told my mom and she was obviously in complete shock. She asked if I wanted to tell my dad or my uncle. But I said no. What difference would it make? No one would believe me. I'm the only female in the family. We never really talked about it again and it makes me feel weird that it was just kind of ignored. I fucking hate him. I feel like he ruined my life without me ever knowing and maybe this affected me more than I thought. But then again, I think, is this even a big deal? Was I sexually abused as a child? I don't know what this means.. Help?