Worried of infertility

So I have been sexually active for about 5 1/2 years. For 3 1/2 I didn’t used any protection at all. (At that time I was only sleeping with one guy) and I know that was stupid now. But I have been on birth control from that point on, first I started on the shot then had nexplanon for a year, and now I am on the pill. But for some reason i have been worried about infertility, just because of my past of not being safe and not falling pregnant (thank god) but me and my SO are thinking about possibly starting a family. I plan on seeing a doctor about my concerns, but I hustled wanted to see what others had to say.