Conceiving after Implanon? Help!

After my son was born in 2011, I went on Implanon for a little over a year. We decided to try for baby #2 so I had it removed. Nothing really changed as far as my periods, and I've always been Really irregular, such as one maybe every 2-4 months? It's always been like that, ever since I first started. We now do absolutely nothing as far as birth control and haven't since I had it removed but I think the Implanon is still in my system. We've seen a fertility doc for my hubby, who really had nothing to say other than his sperm is misshapen?? My ob really can't give me any answer as to why we haven't conceived and I've had Pap smears and tests done. I'm a little over weight, but pretty active, I do smoke, I don't drink rarely at all, and absolutely no drugs. I guess what I'm trying to say is, does anyone have any experience with conceiving after Implanon?