need advice..

so recently, I’ve been seeing an older man. I’m 19 and he’s 39. We met at our workplace. I really really like/liked him. Then.. we had sex.. my ex had a relatively large dick. This guy, it’s very small. And I’m not sure how to feel about that. I know I shouldn’t judge but idk it was weird for me. And he knew me when I was young, like 12. So everyone else finds it weird that we’re seeing each other. Anyway, I fell for who he was. But, now, I just think he’s weird. And he called himself “daddy” the other day and idk if it was sexual or because he has a 3 year old.. but it was through text and about going shopping. How would you guys feel? He’s never been with anyone younger besides me. He’s an open book, nothing to hide at all. So, I don’t think hes in it for the age, but it’s all becoming weirder to me. But, I don’t want to give up on something great because the first time we had sex it wasn’t very good and it’s making me question things