Concern about IUD

I had an IUD (Mirena) for several years, probably about 7 years or 8 years in total. I had it removed about 2 years ago because it slipped and I never had it replaced. 
I have been with the same partner for 4 years. We never tried to get pregnant, but we didn't do anything to prevent it either (after having the IUD removed) and I have not gotten pregnant. 
Now we have decided to actively TTC. We have been for the last few month unsuccessfully. I have had a sneaking suspicion since having the IUD removed that it may have affected my fertility. I spoke to my doctor about my concerns and he disagrees. 
Has anyone had an issue with fertility after having Mirena removed?