Feeling Guilty

My son is 4.5 months old, and even when he was younger I would just feel so guilty about leaving him to go out. I am a stay at home mom, and my husband works second shift. Sometimes when we go one dates we have my mother in law watch our son(usually only when we go to the movies) and she absolutely loves watching him for us and always does when we ask, and i trust her 100% and i know she knows what shes doing. But for me I feel so guilty for having alone time and not spending that time with my son, I've tried to talk to my husband about it but he doesn't understand. I almost feel like I'm abandoning him, and I love my son more than anyone or anything. I just feel like if he's crying and I'm not there he's wondering why his mommy isn't there to hold him or pick him up? It's getting to where I don't even want to go on dates with my husband, or go out with my friends for a few hours without having my baby boy with me.