When can a dad take out a newborn??

Me and my ex are arguing about when he should take her out. She's barely a week old and I'm breast feeding her. I'm not okay with him taking her out at all or have her sleep over his house because he has no crib and I'm scared he might be high one day because of how much weed he smokes, yesterday he came to see her and he smelled like.cigarettes and I got so pist. He is also being bossy and im so close to the telling him to fuck off and take me to court if he wants visitations rights so we don't have to argue. I owe him nothing and he basically treated me like shit while being pregnant, only reason why im doing this if for the baby.