Would you allow it?

My sons father can come see him whenever he wants for however long I don’t mind… he doesn’t have a place of his own, and I’ve never met his family, but he wants to bring my baby with him to meet his other child from previous, which they are always welcomed here.. my thing is you want to bring him around people I’ve never met and his other kids mom which has always been nasty to me and told me to abort my baby… I don’t trust my son going around these people, why can’t he just visit him here idk am I a bad mom for not allowing him “quality time” with dad