Taking baby out

We’ve been to the doctors office & once to the grocery store (on a Tuesday evening, when it was pretty empty) but other than that, we haven’t really taken our baby anywhere cause we are nervous. At what age did y’all start taking baby out? I was aiming for 6 weeks but I am losing my mind in this house all day. He is currently 4 weeks.