This is a problem

Okay y’all so I had a baby in august and my dad bought a house in September to help me out and let me and my son live there. I am still with my baby’s father but he can’t see to get his shit together. I know my dad doesn’t want me with him, what father would. He’s in his late 30’s and can’t seem to get his shit together. Had a little job for two months out of the two years. But he stays with me here most of the time. I guess I am embarrassed when he is here and my dad comes over because i feel like it looks bad. What would you guys do?