Will you baptize your baby?

Serena
My husband and I were both raised Catholic but left the church and are now agnostic/atheist, respectively. His family is still religious and although mine isn't, my aunt still gifted me my old baptism gown for "when the time comes."  
I think our families are both expecting us to baptize our son and will be upset if we don't.  It's not important to us to do it and we don't plan on raising him religiously, but at the same time I don't want to offend or cause our families to worry about his eternal soul or anything.  
Is anyone else in the same boat?  Also, if we did decide to do it, would they even let us if we're no longer a member of the parish?