Still not pregnant

My partner and I haven’t been using contraception for the past few years. We don’t have sex that often as I have a very low sex drive (working on that, though!) but we usually do it a few times a month. I guess I’m just surprised and a little confused that I haven’t fallen pregnant yet. We had the attitude of ‘if it happens, it happens’ but it’s just... never happened. Lately I’ve become serious about the idea of having a baby and I’m ready to start actively TTC. I’m a little apprehensive that we’re going to have problems conceiving, so I wanted to know whether my lack of pregnancy is most likely due to not having sex often enough, or whether it may point to something more serious? Opinions welcome!