should i care this much of what people might say or think ?

okay so my husband is 24 and I'm 20 and ever since we met we have always wanted a baby ... but that feeling has been only with each other. you know ?I really do want to have a baby with him and him with me we live together but the reason I don't have a baby I because believe it or not I'm super scared of my mother knowing I got pregnant. I know I shouldn't feel that why because I pay my own bills and stuff but she's my mom and she is a big influence in my life.... and I don't know what to do ..... I'm so sick and tired of over thinking this I just really want to have a baby and I want everyone to be happy for us ... but i find that hard to happen .... keep in mind that I grew up with family constantly telling me being pregnant ruins your life... I don't think nor feel that way.