Fantasizing about quitting my job

Am I alone? Dont get me wrong, I love my job, my coworkers, and I feel that I do important work. But my husband and I are TTC (month 2) and are thinking we might not be comfortable with my job if I become pregnant, because I work for a non-profit in an inner city environment where there have been multiple shootings within a 2 block radius during office hours, in the past year. Some of these shootings are random, not domestic. I also feel that I would likely want to stay home full time after the baby is born anyway. But how responsible is it to quit your job when you find out you're pregnant? What if the pregnancy doesnt succeed? We can technically afford to go to a one salary household, but it would mean tightening our belts a lot. Anyone else struggling with this?