He used my baby to threaten me

His parents called him up to say they were coming to pick up the baby for the day which pissed me off because they never asked me if it was ok and I only have a few days until I go back to work so I wanted to spend it with my baby but I guess so I let my mil take her which created tension between me and my baby daddy. He left to his parents while I decided to go over my best friends, well he got home with our baby before I did and well he called me very angrily asked where I was at to which I ignored his text then calls me to say that if I’m not home in 10 minutes he’s going to take the baby to his moms house. Basically threatening to take my child away because I wasn’t home