Going over daddy’s house

My little one is about to be 4 month her father and I aren’t together he left me while pregnant and moved into his gf’s house right after.. well he was involved up until her 2 months then he stopped he never really took her to his house he would only see her while I was present but I would do the feeding and change of diapers well he recently asked if he could take her to his house for a full day.. I work so I would pick her up right after but I’m not sure I told him I didn’t trust him and he said to give him a chance.. we’ve never been to court and have both agreed to do this ourselves as civil as possible, but I don’t know his gf if she’ll treat my baby right and she has a 6 yr old so I don’t know if the little girl would treat her right.. aside from the “ he has rights” would you consider letting your baby go for the day to her dads house if you were in my shoes? I need opinions.. I only come here to ask because my family would say no right away but I’m trying to be as open minded as possible