I like him but he’s not my type

I already know I’m going to get hate for this, but please try to give me genuine advice…

I started talking to this guy awhile ago. I didn’t like him at first even though he quickly admitted that he had feelings for me not long after talking. We’ve met up twice and went out to eat with my cousin. (Him and my cousin are friends and we met through him.) I also took my 4 month old daughter with us. He bought me flowers, he interacted with my daughter (which means so much to me since her dad isn’t in the picture.) he even carried her car seat for me. He’s literally the sweetest guy ever. He asks everyday how my daughter is and always includes her when we talk about going out. We had plans this past weekend but he was exposed to covid and wanted to stay away as to not get us sick. The only red flag I’ve noticed about him is he likes to drink a lot. it’s almost an every night thing for him and if him and I do get together I don’t want to be exposing my daughter to that as she gets older.

Other than the drinking, the thing keeping me from furthering things with him is his weight. He’s a big guy, definitely over 300 pounds. I’m not exactly skinny myself but I’m significantly smaller than him. I’m only in the talking stage and have already had family members make comments to me about “dating a fat guy”. I know I sound so shallow, and I really like him personality wise, but I don’t want to get made fun of for dating him. I’m also worried about the sex. I’m not going to lie, I love sex and it’s important to me. He’s already asked me if the size of a penis matters to me and told me he doesn’t want me to be disappointed during sex, so that just tells me he’s not got a lot going on down there.

What should I do? I’ve tried cutting things off with him in the past bc I know it’s wrong for me to think these things, but I missed him so much that I reached out to him and we started talking again. I honestly do like him so much, I just don’t know what to do.