TTC with breast lump? (Non cancerous)

I have a fibroadenoma, have had it for years. Never paid attention much until last year …. Had a mammogram and biopsy. Confirmed fibroadenoma. It’s a small non cancerous breast lump. Estrogen makes them grow so sometimes they grow in pregnancy. Have kids already would like to have another. Would anyone be afraid to get pregnant with a fibroadenoma? Apparently they are very common, affecting many women in their 20s and 30s. Mine has not grown and if it has its very small amount over the past 5 or so years I’ve had it.

I just don’t want it to grow big and fast while pregnant and become cancerous. My doctor said it doesn’t affect me having more children but does anyone have one? I just worry a lot