My family thinks we are too young.

I am 23, my hubby is 21. We have been married for 2.5 years and we feel ready to try. Family has said anything from "you have no business trying for a baby right now" and "you couldn't afford another cat, let alone a baby!" We may not be rich, but we are making it. I feel like parents have made it with less than we have. I'm starting to feel like if we do get pregnant, it will be a bad experience because of family! Advice?