Longish post but need advice *

My husband and I met when I was 17 (turning 18) and he was 21.

We started dating and after 2 months we decided to try for a baby. I was so excited because I've ALWAYS planned on starting my family at a young age. We had the money. We both worked full time and were able to provide for a baby. Just want to throw that in there so I dont get hate messages or something. His sister and boyfriend were trying as well , we made it a plan to be pregnant together. In march of 2017 i got a postive pregnancy test. I ended up having a miscarriage and the next month his sister found out she was pregnant. (She had a beautiful baby girl)

We ended up moving back to my home state 3 months later and we still "tried"

Fast forward to now , husband doesnt want a baby. But he isnt doing anything to prevent it. However he has low sperm count so having a baby cant really happen until he wants to.

I finally had a conversation with him and he said on our 5 year anniversary we can start actually trying for a baby.

Which would be in 2 years. Which I'll take that answer cause it's better than when he said 10 years lol. I wouldve been 31.

Would it be wrong of me if I still continued to try and got all of the right vitamins , conceive plus, started tracking ovulation and bbt?

Should I just respect his wishes and continue to wait ?

It just sucks cause it's been 3 years since we said we were going to have/try for a baby.. hurts you know?