Is it still rape and does it count?

So when I was in 4th grade someone had been sexually and physically abusing me and one time he stuck himself inside me and pulled out like seconds later. I already new what it meant to be a virgin since my family is highly religious and had explained it to me. He said it wouldn't count cause it was short and that it wasn't rape because I allowed him to. But I'm 14 now and I'm just so unsure, I mean I let him but only because I was afraid he'd beat the crap out of me if I didn't! Things have been resurfacing and I need some help!