Excited and guilty about returning to work

So I'm a massage therapist so I get to chose my hours. My daughter is a month old now and I want to go back to work when she is two months old for one or two afternoons a week for like 3 or 4 hours but have got a lot of negativity from friends and family about doing this. I love my daughter more than anything, but for my mental health (and financially) it would be awesome to go back, plus I love my job. (Also, I'm self employed so dont get mat leave, so I haven't been paid in months!). Any other mamas who went back to work pretty quick? :)