What if my boyfriend is ready but I'm not

So I'm 17 and still a virgin, but my first boyfriend isn't a virgin. 
I've always had the hope that the first time I lose my virgintity they would be losing there's with me...but Doesn't seem like that will be the case 
I really like him like crazy, but I'm not comfortable with myself enough to move things along (not that I'm frigdit) I just like to think I have morals and I worry a lot about my body and self appearance and actions. 
So I'm just afraid if he tries to touch me...and I know I honestly just don't want to or that to happen because I don't believe my mentality could handle it. (With my anxiety) so please help me