Father's Day for the father of my son?

It's been rocky with communication and ends up in a fight every time I have visitation with my sons father to have him see his son, and he's asking to see him Alone for Father's Day. He has also denied his son even being His and is already engaged to the woman he cheated on me with during my pregnancy (been apart since March this year) I'm afraid he's going to take him and not give him back at the end of the day -that is my worst nightmare. Should I give him time with his son and pray he brings him home or tell him wait for court (which he made himself, I didn't want courts involved)