Short guy

There's this 33 yr old guy (I'm 21) and he's shorter than me. He's really nice! But I'm not interested in anything with him. He keeps asking me to go for coffee or a drink. And I would, just for a chat. BUT the only reason I won't is because he's quite shorter than me. Not looking for negativity, I just need to know what to do or say to him to stop contacting me. Help?!?