When do you say Enough and walk away...

I've been in an almost 3 year relationship. We have a 7 month old together and my stepdaughter who is going to be 11 in a few months. I love my kids more then anything but their father is just draining me. He is verbally abusive and super unappreciative and just starts fights for no reason, I can't Express my feeling without him saying I'm starting a fight and then when I cry because I feel overwhelmed and I have no one to talk to he calls me a baby and I am having a fit for no reason. He never takes care of our son, I am his caretaker 24/7 while my sons father goes outside and gets stoned for hours after work and on his days off he gets mad at me for even suggesting we do something fun because he is so tired and just wants to sit and get high. I'm just over it all, its physically and emotionally draining me. I dont want to split our family apart because of the kids but i miss being happy.... I can't even imagine losing my son for even a day because his father doesnt ever take care of him, he will just start dating some random girl to take care of them or make his parents watch him and they dont even watch the kids and his mom smokes in the house which I dont think is ok at all but it's her house so I dont say anything. Idk what to do. I'm thinking about talking to a lawyer. I am a stay at home that's my only thing I'm worried about because I dont have a paying job even though I'm the only one who takes care of our son.