Fibroid found!

So I’ve had all my blood work and ultrasound done in which mostly all my reports were on point for me to safely say I can conceive but last week I had my appendix removed in which the post surgery report mentioned one small fibroid in my uterus. Should I be worried as only a few weeks prior to that my pelvis examination seemed perfect. I’ve been trying for a year do you think this little fibroid has been the problem the whole time?