I dont know if want to allow my 2 year old to visit his father

My ex family reach out to me and asking me can they bring my son with his half sister that is his dad other child to go visit his dad at the Correctional state prison cause my 2 year old father told his family that his wanted to see our son I told them that I have to think about it and I will call them back with a answer cause I'm feeling alittle bit uncomfortable allowing my son seeing his father at the state prison I mean I'm just lost me and my ex don't get along that well no type of communication and for 2 I wish he would contact me and talk to me about seeing his son he never call or never wrote me a letter to see how our son is doing like I move on everything and I don't have a problem allowing my son to have a relationship with his father but I wish things was different