Advice?

I’ve been dating this guy for a while, he stays at my house, we make out and he’s fingered me but that’s IT. I have HSV-2 and he knows about this. But I’m not having a breakout, and haven’t in over a year, so it wouldn’t be transmittable, and he knows this as well. He tells me he’s falling for me but he won’t let me even touch his.. man parts. But he fingers me? He tells me he wants to wait. Which I understand but if we’re waiting why can’t we wait on everything? Like he freaks out when my hands go near him, even through clothes. I’m scared he thinks I’ll judge him or something? He tells me it’s gonna happen soon and he really likes me and wants to be with me.. I wouldn’t even care but him pushing me away so much makes me wanna do it so much more 😭 has anyone else dealt with this? I’ve heard from someone he might have ED. I just don’t know what to do. I feel so unattractive.

Edit:

I talked to him about it. He said it’s absolutely nothing to do with me and he wants to wait to be intimate with me because he really cares about me and wants us to work out, and he’s scared if we have sex too soon it wont be the same and it’ll get boring fast. Y’all I’m in love.