Dating question for late bloomers

Have you ever had problems with physical contact ? I’m in my mid-20’s and I just went on my first date in years. The date was great, the guy was great, but it had a lot of casual physical intimacy (hugs, hand holding, even a kiss) and I was weirdly uncomfortable about it (I also want to say I initiated half of it so it wasn’t like he was crossing boundaries). I literally had to sit in my car afterwards and hug myself because I felt like I had to counter-act the date touching with MY touching. Is this normal for people who don’t date a lot? I’ve never had problems like this with friends or family, I’m actually pretty physically affectionate. I like him and I want to continue what we’ve got going, how do I handle what’s going on?