Need advice

Boyfriend and I have been fighting and he always says negative things thay always lead me to asl him why are you even with me then and once again he tells me honestly idk why I am then proceed to tell me its bcs of my son (from a previous marriage) and says hes scared to lose him! If that isnt a slap in the face idk what is. But then he will apologize later and says sorry you know i love you and know I wanna be with you blah blah blah. Its to the point idk what to do. My sons father is in prison and my boyfriend has stepped up and been there for him. But how do ypu even get tue courge to walk away without hurting your son. I'm broken from hearing this for the second time.