Endocervical polyp

Just wondering if anyone else has experience with this. My ultrasounds show an endocervical polyp (polyp growing inside my cervix). My RE told me it was no big deal, to leave it and that it's not a barrier to TTC. My family doctor on the other hand thinks I should have it removed as there is a chance the cells are abnormal (my RE won't do the procedure). After my first round of fertility drugs it grew. So now I'm waiting for a Gyno referral to have it removed. I am so tired of all of these things that put treatment on hold (I also developed ovarian cysts after my first round). Anyone go through this? Any ideas on recovery time?