Not telling Family

I had a chemical pregnancy last week. DH and I decided to not tell family because everything happened so quickly. 
Yesterday my brother in law asked me what are we waiting on to have some babies! He said I'm taking too long and they (him and my sister) have baby fever!
All I could think was "I'M TRYING!!!!" Lol. 😩😩😩