Relationship Advice

So I like this guy like a lot, he's 28 I'm 23. The age difference is not really that big and not a problem from me at all rather than the fact that he has a kid. Back story: his 2 year of college he went back home and hooked up with this girl which ended up getting pregnant and doing holes on the condom to make him stay with her, she also didn't tell him that she was pregnant until she was 3 months in. He dropped out, moved in with her to raise the baby together, they had the baby she kept going out and partying while he was taking her on the baby. When they got into fights she would punch him (he has scars) until one night it was too much and he put whatever he could fit in his car and left. Anyway I've been talking to him and we been hanging out and we really enjoy each other's company. He says that he wants to finish college and get a job, he has ambition in his future but right now he can't because he's trying to get enough money. We have talked about being together and before I didn't see myself with someone like him but now I do, I really like this guy and for the first time I feel and care about a guy so deeply unlike my other previous relationships. My question is: is this weird? am I too young or stupid to fall for someone that has a child and dropped out of college?

I have already heard "oh wow she's too young to be a stepmom" or "why are you with him, you could do so much better" but honestly I don't care that much besides the kid is not with him 24/7 rather than 2 to 3 weeks during the summer with him, and as much as I don't mind the kid part I always wonder is it going to be hard? what's going to happen if we become more serious and I meet him?

Thoughts? Advice?