I'm scared

I have been with my boyfriend for 3 years now we have sex with no protection and he cums in me and I never got pregnant but the doc said I'm fertile. Before this boyfriend I was with a guy for two years never got Prego but when we broke up he got with this girl and bam she Prego I'm between the age 19-30 now not gonna put my age don't wanna be judged can someone give me advice am I not doing it the right time I plan on getting married with my boyfriend currently  and I hope we could have a family