Fibroid & IVF

Long story short.... I went for a polyp removal a couple of weeks ago only to be told it wasn't there and was just the shape of my uterus. Went to see the fertility specialist and he said it's a small fibroid. I've been given the choice of either getting it removed and waiting even longer for my embryo transfer or just going ahead with the transfer. He said he wasn't too bothered about the fibroid as only 0.5cm and usually remove them at 3cm. I want to just do the transfer as I've waited so long now but I'm also scared it won't work because of it. Any opinions greatly welcomed or if you've been in a similar situation x