Is it rude if I don’t text him?

I’ve been talking to this guy for about two weeks. We pretty much text and talk on the phone nonstop. Last night, neither of us had any plans so he came over. Before he even came over, he told me that his back was sore from work and wanted to make sure I was okay with just cuddling. That was of course fine with me. The minute he got here, he was all over me. He was nice, but he was still super handsy. Every time I’d break away from a kiss or scoot away from him, he’d pull me back. I’m not saying I was taken advantage of because I never told him to stop, but I feel like it must’ve been obvious that I wasn’t into it. But he was just pushy. Eventually he ended up pulling me on top of him and we had sex. It wasn’t great, he came and I didn’t. And after, he rolled over and fell asleep in my bed. Around two, he woke up and the same thing happened again. He got super pushy and handsy, he pulled me on top of him, we had sex, and he rolled over and fell asleep again. Around 4, I woke him up and told him he needed to go home. We haven’t spoken since, and I really don’t plan on us seeing each other again. He texted me once a few hours ago, but I really don’t want to respond. Am I an asshole if I don’t answer him? It really just creeped me out how pushy he was.