Choosing not to go back to work... regret?

Has anyone chosen not to go back to work and stay home with their baby and regretted it?

I hear a lot of stories about people who regret going back and missing out on this time but I’m just wondering if anyone has ever regretted giving up their career.

I love my job and it can be fairly flexible, it is just a lot of stress and I want to stay home with my baby! I’m just worried I will regret giving it up and not having an outlet for mental stimulation and communicating with adults regularly.

We can afford for me to stay home but once I leave my position I will never get it back and it will be tough to get even close to where I am now after time away.

Thoughts?