I don't know what to think

I have been birth control free for almost 2 full years. We made this decision together, not because we wanted to have a baby but because birth control screwed up my cycles badly and I ended up being so screwed up, having a heavy period for a month. I was just done, so done. But I wanted him to know and be okay with it because of course, without birth control there's risks for babies. Which at the time we had been together for 2 years and both agreed that if it did* happen it would be fine. Which we basically thought it was going to happen with the way we were "avoiding" 
What I have been doing to avoid pregnancy is, I track my cycles and avoid sex while ovulating and he pulls out even when I'm not in my fertile week. He also pees between rounds to prevent precum with sperm in it. 
Everyone's like "you're asking for a baby when you do this" and we still don't have a baby. I just ended my last period a few days ago. 
Does this mean I'm like infertile or something? We both like, accepted that it was a massive possibility but we haven't even had a single scare. I mean, what is the point in birth control if this is as affective as it has been for us? Does this imply that it will be hard for me to get pregnant when we try? I've wanted a baby since I was 16 but I knew it was a bad idea lol and now I'm 23 and he's 25 and we are thinking about it and I'm nervous about trying because I've been told over and over that if you're not on BC you WILL get pregnant and I'm not. I know it's stupid, but I don't want to let him down if we try and it just doesn't happen. I want to give him children. I'm sorry if this is stupid.