Feeling sad

My husband and I have never not been trying and we are going on 6 years together 3 married 
Lately he's been telling me that if we didn't have a kid he'd be totally ok with that and he seems to happy with it just being us. 
On one hand ok
But on the other hand we love each other and I've always wanted to be a mom when the time was right 
Should I just be patient and let nature run its course?