How do you stop "trying"?

Sarah • 27. Married. My son was born 2/22/16 and his little sister 7/5/17
This gets me every time! My hubby and I have been ttc and people keep telling us to stop trying and it will happen- I know they don't literally mean stop trying, but more stop WORRYING; and stop desiring SO much. I can't help it though, I want a baby! So how don't just turn off my "I want a baby" switch? I've tried picking up different hobbies, God knows we work like crazy. We are even closing in our first house- you would think the exfiebt of THAT would overwhelm me and distract me a little but it doesn't. Have any of you ladies struggled with just letting go and letting it happen "when it happens" ? What Helped?