Fertility is the question!

Hi all! My names Eryckah! I'm kinda new here and I was truly hoping I wasn't alone in this. I've been TTC For a few months. It started when I thought I was pregnant because I had been missing days of my birth control and my period was a no show for a few days. My period has always been off. Since puberty. The first year of having AF, She only came maybe 4-5 times. But I was told that it takes time to regulate so I didn't worry about it. And year after year I only got my period about 3 to 5 times. Sounds fabulous right? I loved it tbh. But then it just didn't feel right to me. I wanted to get it every month like my mother and my sister and all my friends did. So I went to my doctor and she put me on birth control. But I was only on it from June 2016 until March 2017. So not too long. Quitting wasn't bad at all. Though I stopped in the middle of the pack. I got my period April 9th, and it lasted the regular 5 days. But all the while my SO and I started trying for a baby. And when I didn't get my period I got a bit excited but my tests were negative the entire month of May. And I got my period June 4th and it ended around the 9th. I know that my cycle won't be regular when I first get off the pill but I'm afraid that since I never had a regular cycle, if that could mean that I may be infertile or just less likely to get pregnant and wondering if it would be a good idea to go to my doctor and get a fertility test. I just need some sort of advice. I am anemic if that plays a role in it, I'm not sure. Any and all advice would be WONDERFULLY appreciated!