Trigger warning: Rape (this story isn't about rape but it mentions it)

I recently last (a week ago) got into a relationship with a guy who I met in January he's 21 and I'm 19. We've been close friends who do stuff together for a while (e.g. we go running and swimming together every week and often go cycling with some other mates too). Last Saturday I went around to his house for the first time to bake cookies and he kissed me (which was great!). Since then we have been doing the normal running and swimming but hanging out afterwards and making out.

He asked me if I wanted to go around to his house and stay on Friday. We've talked about it before and he knows that I have a 4month rule when it comes to sex. In the past (because I was raped when I was 12) I have found it super hard to move fast in relationships and I couldn't stay at other peoples houses (even for sleep overs) because I would get so anxious because it was on a sleep over that I was raped.

I was thinking to myself that I don't need to feel anxious as I am now a grown woman who makes her own decisions and who has full control over her behavior.

However I got to his place and I was super anxious but I kept it under control. I told him how I was feeling and he was supportive. However later on in the evening we got talking and he said he felt like he was being rejected sexually as I kept stopping his hands from touching my nether regions. That really resonated with me as in my previous relationship I felt really sexually rejected so I let him touch me and I enjoyed it, or at least I told myself that I enjoyed it.

To be honest I wish we had moved more slowly. I don't think he really gets what a big deal it was for me to even be at his house. I did it to prove my anxiety wrong and because I enjoy spending time with him I'm not really sure whether I want to spend time with him anymore and I'm freaking out because I feel like I've made a huge mistake.

What do you think? What should I do?