Dad Visits

My baby is just 2 wks old. Dad & I split. I've been trying to be fair and as flexible as I can with him visiting her. We had a pretty recent horrible break up but trying to be mature I still allow him to come to the home we once shared to visit his daughter, the only thing is I have male friends and things I'd like to do {i.e. run errands, go out to eat, go visit} he's also dating and doing his own thing, he doesn't have a place of his own yet so I can't just take her to him for visits and I'm EBF. He ties my whole day up saying he's coming at this time and that time because then I'm sitting here with baby waiting on him. I don't wanna be a bitch but I kinda wanna give him a time frame that he can visit. Is that a bad idea.