Insensitive family rant

Been trying for 3 years. And when my family brings up why we don't have e kids yet a i start talking about it, their response is always stop trying so hard and it will come. I just feel that is the most insensitive thing to say to someone that wants children. And my husband and I desperately do. I know constantly worrying About it and stressing over it doesn't help. But I just hateee when anyone says that to me...anyone else have this happen to them.