STRESS- does it really matter?

Both my husband and I are a couple in our 40's TTC. We run two businesses & care full time for my high care elderly father-in-law who has dementia...
​Some days it gets too much for me trying to work from home and look after my father-in-law. I was told by a doctor recently that stress doesn't matter but when my DH and I were too worn out to BD on my peak day this week it broke my spirit. Common sense has to tell me that stress DOES matter. What do you think? Thank you... Mrs Discouraged