I'm ready but worried what everyone else will think

I feel like I'm ready to have a baby. I'm 23 and my husband and I have been together for almost 5 years. We recently got married and will be celebrating our 1 year this Oct. He's not exactly gungho about having a baby right now but he's not trying to prevent it from happening either. My family and in laws would be ecstatic but all my co-workers keep telling me I need to wait and I'm almost to the point where I don't want to get pregnant because I'm worried of what they will say. Every time someone asks me I lie and say I don't want to have kids anytime soon but in my heart that's not true. My husband is about to turn 28 and we are financially stable. I don't even know what I want right now. I'm so worried of what people are going to say but should I be?