Leaving the house with my 2 month old scares me!

I have yet to leave the house with my son by myself. Just at home if he misses a nap he gets so fussy! I can't imagine taking him out and he fuss out at Target or something. And I feel bad disrupting naps. I normally wait till he wakes up ON HIS OWN. Feed. Jump in the car. Hoping to be back home or at Grandma's for his wake time so I can try to get him to nap there. Am I crazy for being so focused on his naps? My husband thinks I shouldn't make plans around my baby's naps. If I want to make plans with someone I should and just wake our son up. How do other moms do it?