coparenting. am I wrong?

Baby father and I have been having calm talks about our son. He wants to have 2 days where the baby is at his house with him and his friends/girlfriend and I said that I won't leave my son til he's about 2 where I know he can be looked after properly. 
He then begins to call me jealous and that it's his baby he can do what he likes and he will take ME to court. 
I can't see why I should leave my baby's side and bring him back and fourth every 2 days instead of settling to an environment but opinions? 
Edit: I have told him that he can visit by me as long as he likes with his girlfriend if that makes it better. I just don't want to leave my baby for every other 2 days and especially with no financial support from him. The baby will be born July.