Sex and physical relationships?

I’m in a serious and committed relationship. We’ve been together almost 3 years and he was there for me when I was raped. My boyfriend is extremely loving and understanding and wonderful. I’m worried about things getting too physical though. We take things slow and he always backs off if I even begin to push him away and he asks if something is okay before he does anything. I can kiss him but I can’t really handle anything else. I feel like I’m just being ridiculous about things and that it’s not fair to my boyfriend. The one time I mentioned it he assured me it was okay and we didn’t ever need to have sex. I still feel like a disappointment and there’s something wrong with me though. He’s starting to plan a life with me though and I don’t know how anyone could be happy spending their life with someone who will never be physical with you. When you’re older it can make sense but he’s 20 and he’s willing to give that up. He even mentioned adopting kids instead of ever having his own so we could still have a family even if I can never handle sex. How hard was it for anyone else to have a physical relationship? I’m just looking for some reassurance that I’m not the only one it’s hard for and that it is possible to do physical stuff again. I feel a little crazy over this whole thing. Especially since it’s literally the middle of the night right now but I can’t stop thinking about it.