Baby Fever

My husband and I have been married for 6 months now, and I would really like to start trying to conceive in a year or less. The problem is my husband would like to wait 2 or 3 years. I try to talk to him about it and he always seems defensive and wants to change the subject. I have a history of ovarian cancer and had an ovary removed. Because I have half as many eggs as an average woman, my doctor always recommended to not wait too long to have children. I try to explain this to my husband and he always says Its fine and we won't have any trouble when we try. Has anyone else ever been on different pages with your SO about having a baby? And what helped make them more open to talk about it and see your side?