Grow up!

I just got in an argument with my boyfriend because he acts lazy when it comes to doing shit. He doesn't have a job, and his baby mother doesn't allow him to see his daughter. I suggested that he can still go to court to get visitation rights but he says no because they're going to say he can't have his daughter because he doesn't work. I've been with him for a year and it's irritating me to be with someone who doesn't put in effort to see his child! He lives with me and my child (from a previous relationship) and he pretty much is hands on with my son. But yet when it comes to him and his baby mom, he gets mad when I make suggestions. Then he tells me to stay in my place. I work two jobs to support us and all he does is smoke, sit at home and watch tv. Literally all day or go play basketball. I can't take it anymore. I need a man that's willing to step up and be a man! I'm tired of his excuses. I can't plan a future with someone who isn't willing to grow up. Am I being too harsh? I really need advice,