How do you answer being asked when your having a baby

My husband and myself recently got married in February, ever since I’ve continuously been asked when I’m having a baby. Is it just me or is it actually rude to ask someone when we’re gonna have one, how do you respond to this question? Cause I’ve just stopped answering now.