my bf threatens me to take our baby from me !

Since the day we fought our baby boy home I've been the one taking care of him and I'm the one to get to every night and everyone he fusses..I get no help from him and he doesn't make no time for us ..he comes home late everyday ..I told him that it's better if me and the baby leave to my lols and stay there so she could help me out since he's never here and al he tells me is that I can't take the baby and if I do then he'll take me to court