Do you think I’m being rude and wrong for doing this?

My daughters father and I haven’t been together for 5 years broke up when she was 1 years old, well his been going back and forth between his parents place and who ever his dating, well he just got kicked out and has to move back with his parents. so I told him our daughter won’t be going there for his weekends (don’t care if she went just for the day) his niece, her boyfriend and son also live there and it’s only a 2 bedroom and he thinks I’m the one being wrong for keeping her from him when I’m thinking of her well being.