It scares me

I am 22. Me and my boyfriend have been together for almost three years we very rarely use protection when we have sex and I haven't gotten pregnant. I know I'm still young and I've had lots of people to tell me that we need to wait and enjoy life before having kids but I do want children. Whether it be now or in the future. But I'm wondering if there's a chance one if maybe there's a change one of us could be unable to have children. Any thoughts?