Would this be considered rape?

Roughly 6 months ago I was with my friend with benefits I guess you could call him. We haven't put titles on anything. Anyways, we were kissing and stuff and then he kept trying to take my pants off and when I said no and told him I was a virgin he asked if I was saving myself for marriage. I told him no that I just wasn't ready to have sex that night. He said okay and then suggested that we dry hump. I asked him if he was sure about that cause I didn't wanna make this worse for him and he said it was fine. After a bit of that he flipped over and told me to go on fours and said "I'll keep my shorts on; don't worry you'll like this". So I said okay because I was curious what he had in mind. He goes behind me and I feel him but wanting to think of him as a better person I didn't really question him besides asking again if he had his shorts on still and he said yeah. That was a lie. When he drove me back home I knew that was maybe rape but I wasn't too sure since I told him "not yet". Yet being the keyword I guess. Since that was my first time and I was indenial I thought maybe it's just because I wasn't prepared. So then up until last week we've been having some sort of sex for about 2 times a month since that night. It wasn't until recently that I really started questioning it again. Was it rape the first time since I said no and was under the impression that he still had his clothes on or am I overreacting for nothing??