Am I A Horrible Person?

So it's Valentine's Day right, and I was supposed to hang out with this guy I had been talking to for maybe 3 weeks. We aren't dating but we have been flirty. And I was really excited until I saw where he lived. It's in a really bad part of town and I'm really too scared to drive down there. Am I a bad person for thinking that? Is it wrong for me to try to get out of it because it's not like I'm in love or anything. I have nothing against him or his family but it's just he lives in a really bad place and I don't want to go. What is a not bad way to tell him that? I feel really really really bad like he's gotten me gifts and I'm over here like not wanting to go because of where he lives. It's really upsetting me