Going out WITHOUT Baby

My fiancé wants to go out on a date today. He's noticed I'm a bit stressed and crying constantly with the stress of being a ftm and getting used to our sons schedule (lack of sleep).
Our baby will be a week old tomorrow. I'm FREAKING OUT! I appreciate it, really want to go out but, my little one is so tiny :( 
He wants to leave him with his mom and was planning on taking me to dinner and a movie.  
I just can't stop thinking about the what ifs and don't think I can bare to leave him for a few hours :(. I don't think I can do this
Please what do I do ladies?! Has anyone been through this? Is it to soon to leave baby?