Should I be concerned?

My man and I have been together for 4 years now... and it’s not that I’m trying to have a baby but we definitely aren’t trying very hard to not have a baby. When we first started dating we went about 3 months doing just the pull-out method. But my mother found out and put me on birth control for about 2 years and recently within the last 7 months I stopped taking it because I just kept forgetting and it also made me gain weight. My period has been really wonky but the entire time I was not taking my pills my man never pulled out. He would cum in me every single time and somehow I’m not pregnant. Could it be that one of us is infertile? I don’t want to bring it up to my mom as she would just get grossed out and pissed for me not taking the pill