Husband has his panties in a bunch

Fellow Moms, I have a friend who lives in SC, she has 2 kids. She recently bought a house and moved there with her husband and kids. We have known each other since 3rd grade and are now in our thirties, so we are close. A couple girlfriends and I want to plan a trip down to visit for a thurs-Sunday trip. My husband is freaking out, and it's sort of making me feel guilty, which I shouldn't. I am a stay at home mom (I babysit kids to earn my income, so money isn't the issue) I rarely get out to do things. I need this refresher and I don't think once a year is a lot to ask. Should I feel guilty? Do other moms go on short vacas with fellow moms or friends?