Scared?

Is anyone else ttc but actually kind of scared to get pregnant? Maybe it's just the timing atm. My husband and 6 year old and I are currently living with my parents again. We are still trying to have a baby. I've had 1 pregnancy scare since we have been living here but this time instead of being really happy and excited I got overwhelmed with anxiety about how mad my father might get if I got pregnant again. Or how my mom would react. Should we just keep trying and not care what they say because it's my family and my life and I want more kids or should we just wait?