Am I wrong???

I just had a baby who I'm very protective over. Me and the father of my child got in a disagreement because he feels the baby see my family more than the baby see his family. I never told his family they couldn't see him. As a matter of fact every time they ask for the baby I bring him over. Furthermore, going to my mom's house is beneficial to me. Sometimes I go wash our clothes, or if I need someone to watch the baby for an hr or two I'll go ask someone to babysit, or if I wanna take a shower, or do my hair without being interrupted I'll go to my moms. I never stay the night tho. My problem is everytime I take the baby over his mom's house they ask us to stay the night. It bothers me because we was never allowed to stay the night before baby (not even her own son who was sleeping in his car for the longest) but now that the baby is here she always wants us to stay and only because she wants to keep the baby, and he's breastfed so he can't really be without me right now. Anyway I told my boyfriend I would stay if he wanted to stay but he could tell I didn't really want to. So we got into a disagreement and because I didn't really seem happy going to his moms. Him and his brother is trying to paint this picture that I keep him from his family and only allow the baby around my family when that's not true. I'm comfortable at my moms, my mom respect my wishes when it comes to the baby meanwhile it seems like when I go to his moms house she doesn't necessarily respect what I say when it comes to my baby. I need opinions. Please someone tell me if I'm tripping or not.