Just don't know what to do

I love my kids father more than anything... This who I want to spend the rest of my life with but he just don't want to grow up, we're 22 with 3 kids... We got our own apartment but his dad stays here also and I decided I'll just move out and get my own place, he can come and let his daddy have that apartment and we work on been a family or I don't want him to see my kids no more... His daddy 42 years old I feel he shouldn't be living with us. Is it wrong if I just pack up and leave????