Sleeping on back

I know it is not recommended to sleep on your back as it can potentially be harmful for the baby. I asked my doctor about it and she kind of laughed it off saying “I think sometimes people read too much and get too obsessive about things. You need to be comfortable to sleep.” I find myself on my back a lot, without even meaning to. Has anyone talked to their doctor about this issue? Wondering if there is a specific time in the pregnancy when it’s more important not to be on your back