"Too soon to have a baby"

Lisabet
How many of you have heard this? And what was your response? 
My husband and I got married on June 11th 2016 and we're trying to have a baby. 
He already have three kids, two of which I've raised so it's not like I can do crazy "non-motherly" things. Not that I would want to anyways..
How should I respond to my family telling me it's too soon to have a baby? :(