Family hurting my feelings...

My family keeps pestering my Sister in law to have another baby. She has 1, he is almost 3. She then points to us and says "you better start talking to those two." Almost everyone's response is, "No, they aren't ready for kids yet!" Little do they know we have started TTC. This really hurts my feelings that they think we aren't ready to have kids. Are they going to feel the same way when we do get pregnant? Am I crazy for letting this hurt my feelings?