Bd wants to take me to court

My daughter is one week old we are not together like that but he wants to have her at his house with his family without me present there and I refuse and tell him that’s not right. I’m not breastfeeding because I have to wait till the doctor says it’s ok but still I think it’s wrong that he wants to have her because she’s so small , I even tell him he can come visit here with his family but he doesn’t like the word visit he thinks I have too much control of the situation and actually wants her at the house instead of visiting. we never agreed on this but of course a narcissist like him will say other wise he has his family backing him up and idk what to do

*Update* so if you wanna know how many people came down to visit my daughter on his side the family I’ll tell you, it was his dad, his brother, his step mom, his sis his mom and his sis friends, so many people have carried and kissed my baby and I as a mom wasn’t able to say anything. My daughters umbilical cord fell off the sis kept it and never told me until I realized it fell off. I haven’t said anything to not cause problems, I’m currently staying here with them for a few days till his dad and bro leave back but this is ridiculous I should be able to have a say in things. And for those of you saying he should have 50/50 or he should take me to court, I never said he can’t be with his daughter I just said she’s very young and that I’d rather wait till she’s a few months for her to be away from me.