Baby daddy

Ok so i recently started letting my sons father have him over night hes 2 but because he never was there for him or showed me he could take care of him i never trusted him to have our son alone now he will be 3 in may and his dad wants to take him out of state to see his family, yes our son should know his dads family as well but i have a hard time trusting he will bring him back to this state and that he can handle having him full time while hes there it honestly scares me to let him take our son for that long. He still parties and is irresponsible. Please help me i have no idea what to do