Should I stay or should I go?

I've been unofficially seeing this guy K for a while that I was in a relationship with about 2years ago, we were both only 17&18 at the time and things were amazing until I miscarried his baby. We lost contact until recently and realised our feelings for each other were still there. Only thing is now I'm a mum to my gorgeous 5 month old and went through a lot in the time I was with his father M. To be brief M was violent towards me from 6months pregnant to the point I was hospitalised and was again beaten by him yesterday (he turned up at my house raging because I'd been seeing my K with our son in the house) I ended things with M nearly 2 months ago but he's still harassing me (police have been involved) K has helped me so much build up my confidence and though we aren't together we've been having sex ALOT! He's amazing in bed and better than anyone I've had before! Though we are unofficial he's recently moved about a 4hour drive away (I don't drive) he's asked for me and my son to move up there with him so we can build on our relationship again but I don't want it all to be about sex and now my babies dads threatening to get my son taken Away and finding any excuse possible for me not to go. What do I do because I'm completely in love with K but it's only been 2 months since M and he knows I never loved him as much as K. What do I do? Go or stay! HELP!!!