Need some advice

I need some advice might be a lil long and a lil confusing I had a daughter 7/28/15 and lost her at birth now I have a son born 8/5/16 and his family on his dad's side want to keep him over night. I'm not for it because dad says he is our last and it scares me because I lost my daughter and I'm guess you could say scared to let him go yet the thing is how do I tell them no and not have them or my so mad at me for saying no. It's like when I am ready for you to have him over night I will tell you. It's like ya my family lives down the road and they haven't even asked to keep him over night. Should I let him stay over night or not at 3 months?