Can they take my baby away?

I've had problems with my boyfriend for a while we had our first baby 4 months ago now he's saying he doesn't want to be with me, that's cool, that's fine, I wish I had stopped having sex with him instead of bringing our baby into a toxic relationship, but he's saying HE'S keeping our son? Excuse me? So your breaking up with me and you want me to just leave my baby? The fuck....his mom agrees with him "if you to aren't together the baby stays here" I live with his family have since I was 17 I'm 29 now, we were all supposed to be one big happy family, is it possible he can take him? I'm a good mom, I have a job, I don't drink or do drugs, I have a place to go when it's all said and done...we signed an affidavit of parentage in the hospital so he will have to take me to court for custody right?? I should of added he doesn't do shit for our son he's one of those wake up late and play video games all night when he's not working baby daddies🙄🙄 CHOOSE YOUR BABY DADDY WISELY LADIES!