2nd baby infertility

I have a 5 year old who is a blessing! My husband and I are both young and want to have another baby, he is 22 and I am 22 about to be 24 and I know most people don't think we're ready but were ready. We both have steady jobs, he's a soldier, and I work as well. We make a decent about of money monthly and we just really want to have another baby before it's too late. I am told the older you get the harder it is to get pregnant. We've had safe sex up until 6 months ago and it just seems like I'm never going to get pregnant! I don't want to seem selfish because I know some women have been trying years and haven't even had miracle babies like mine, but I need to know that I'm not alone, and there is others having the same problem! Just for the sake of being normal.