Would keep a job that pays well with good benefits but makes you miserable?

So my husband and I are trying for our first. I have a very well paying job and my benefits are outstanding. But the stress and emotional/psychological toll is very high. Originally we agreed I would stay until going on maternity leave then take that time to find a more meaningful job. Recently he changed his position - pointing out the pay and benefits are too good and I'd be better off staying and trying for an internal position that's a better fit. I get with a family you need to make sacrifices but can I sacrifice my mental well being like that? I will also add that, aside from our mortgage, we are debt free and very careful to live well within our means/under budget. Do I want our children to grow up with the idea it's okay to be miserable as long as the pay and benefits make up for it? Or that it's okay to make less money as long as you are doing a job you enjoy?