Scared about infertility

My husband and I have been married for going on 5 years this July. We still don't have any children and recently we've been talking more about starting a family. I'm 24 years old and he is 28. We have never used protection the times we've had intercourse and now that we are actually planning for a baby I'm afraid that I won't be able to conceive. Is it weird to feel this way? This week is my ovulation week and we've been consistent with having intercourse. Any advice/ tips help!! Thanks!