Bd and his sis wants to take me to court

So my daughter who turned 3 weeks old today I feel is too young to have overnights at her dads right now, we are not together and we live separately, when I was pregnant we agreed on it being 6 months that we start trying the overnights but as soon as she came out of the hospital he switched it up on me, his sister ended up making a schedule for us were he had her the whole day on Sunday and I had her the whole day on Saturday, I never fully agreed to this but because I was scared to cause fights I said ok. We tried the schedule out last week, I’m still not comfortable with it, as a mother who’s breast feeding and the fact that I just gave birth I think it’s messed up that people don’t see the selfishness in that. I never once said he can’t be with her, have all the time you need with her at his house without me there for a few hours but overnights no. His sis and him are basically attacking me telling me they don’t see the problem in that and why do I gatta have more time with her than he does. They claim they’re not against me but I know they are. For the weekdays the schedule is they pick her up at 8am and I go get her at 3pm, the sister doesn’t leave work till 5pm and had a problem with the fact she won’t be able to see her niece like it’s my problem... this whole family is filled with drama. The only people I have to back me up is my sisters. I guess at the end court is what’s gunna happen Irregardless