Is it weird...?

So I’ve been talking to this guy that works for the same company as me but on the opposite side of the state (we worked together once or twice while helping out that office) and things have started getting heated and romantic between us. The only thing is he happens to be nearly twice my age (I’m 23 and he’s 40) and he has a son (12) and I’m planning on moving across the country soon. I really like him and he’s made it clear that he likes me too and it seems like we’re both willing to try making things work when I move. I just know that people will be weird about me being with a guy who’s that much older and has a son that age.