Question about the Depo Shot

I had the Depo Shot every 3 months for 4 years straight. Never missed a shot.

I quit birth control completely about 3 years ago. I've heard that it takes a while for any birth control to completely leave the body so my husband and I waited until last year to try to have a baby. Long story short, we still haven't gotten pregnant. I know the Depo Shot can cause irregular periods. I only had maybe 2 periods over the 4 years I was on the Depo Shot. Since October 2015, I've had periods every month and they have become more and more regular. I figured that was a sign that we could get pregnant. I also know that the Depo Shot has a lot of side effects but I didn't think it would take this long for the birth control to be out of my system. Right?

Can the Depo Shot cause infertility?