Visitation

Me and my sons father has been split up for a year. We broke up around the time he was 6 months old. Out of the whole year he came one time to see him, and has probably seen him a total of 9/10 times due to me bringing my son to see him. The last two times he told me he was coming to visit he never showed up. He texted me a couple days ago wanting to pick him up and take him to do something. I explained I didn’t feel comfortable due to the fact that my son doesn’t really know him and that he needed to get use to him first before being alone with him. He proceeded to tell me he has a new girlfriend and he doesn’t want to co parent anymore, he would rather get him alone. He’s never been legitimized and has never took me for visitation which he says he’s going to do NOW, after a year. Do you guys think I’m being selfish to not let him get him alone? Our relationship was very toxic and he was on drugs durning it. Which he’s clean now but I still have trust issues. I tried ti explain it wasn’t because of his girlfriend but the fact that he hasn’t came around in a year other than me taking him to see him. I’m fine with going to court but as much as I hate to say it I don’t want him to get him until he’s able ti talk to me and tell me what goes on at his dads house.