When are you going to have kids?

I’ve recently been asked, by multiple family members at a gathering, when me and my husband are planning to have babies. I usually just smile, laugh it off and say, “whenever it happens.” I usually don’t get offended when I get asked but, I got a little flustered a few nights ago when I was asked that. I thought, “what if I couldn’t get pregnant? Do I tell them that I’m infertile?” I just got off birth control (ten years I was on it) about two months ago and we are ttc. We’re not telling anyone and we don’t plan to until it is a for sure pregnancy. I guess my question is, how do I respond to that question respectfully? I don’t want to come off rude or irritated, because it doesn’t bother me but I know it bothers some.