Trying to Figure out the Right Time

I'm 21 and recently married. Since my husband and I are both young, we plan to wait at least 3 or 4 years to start thinking about kids. However, when I was 14 I had ovarian cancer and had my left ovary removed. Lately I've been thinking a lot about my fertility and wondering if waiting is the right thing to do. I graduate nursing school in May, so we will have the financial means for a baby, but we won't be as "well off" as my husband wants us to be when we decide to have a child. Does anyone have any comments or stories relating to my issue?