Fibroid and pregnancy?

A year ago, I found out I have an intramural fibroid on top of my uterus (which means it’s in the muscle). About the size of a golf ball. It’s been confirmed through MRI and hysteroscopy that it is NOT projecting into the cavity, but it is distorting it since it’s pushing on it. I’m 8 weeks pregnant (strong heartbeat yay!) My doctor knows all about the fibroid and at my prenatal appointment he basically said that I would just have to come in more frequently as the baby grows to ensure it is growing properly, etc. I’m so nervous about this damn fibroid screwing me over, especially something like a second trimester loss...that would be devastating. I actually spoke to surgeons about removing it and then found out I was pregnant so obviously not touching it now. The uterus stretches with pregnancy obviously so won’t the space stretch to accommodate the baby anyway? And wouldn’t the fibroid be pushed away as the uterus stretches? Looking for some comfort as I’m very nervous about this.