Mirena IUD

I had mirena placed after my daughter was born, had no issues for a little over two years then I started getting heavier, more irregular periods and much more painful cramps. My SO and I had decided to TTC so I went to get it removed and it was gone! I had both internal and external ultrasounds done as well as a pelvic exam and the gyno said it probably came out on it's own with my period, which she said happens often. Fast forward, its been between 3-5 months (I can't remember exactly when it came out, lots of changes in my life right now), and no baby yet. Has anyone else had issues conceiving after the mirena comes out on its own? How long did it take to get pregnant after, and are there any suggests? My SO thinks it's him (my daughter is not his), but I'm worried that the iud has affected my fertility.