Relationship probs

I have a 1 year old baby. I love my boyfriend more than anything we’ve been together for 2 years & 6 months now... but we argue constantly. We live with his parents & family so everyone here’s tonight he made a sense & said that I said he was a horrible dad.. I never said that but I did say “he didn’t know what being a dad was about” bc he’s never home. He needs to grow up & help more. I’m so embarrassed & want to leave but it’s so hard because I love him he’s all I got & I can’t leave.