After the Mirena removal

After my first pregnancy I got the Mirena IUD. Had it in for a little over four years and recently got the IUD taken out in May of this year. Since then my SO and I have been considering pregnancy. We've been going at it unprotected every time since May. It's now about October and we have no luck with pregnancy. Could the Mirena have caused infertility? How would I know? Should my SO see a fertility doctor? Or should I just keep waiting?