Need advice

My husband and I have been together for 2yrs. I have 2 children from a previous relationship took off my iud in novemeber to ttc my 3rd child (his 1st) but was a little nervous at first so got straight onto the depo shot. Stopped taking the depo in april and nothing has happened yet. I know it's still soon but my family always jokes that we are so fertile we get prego just by looking at us. So i figured this would happen fairly quick. Idk if this may be a problem with me from the birth control or maybe with him. He takes steroid shots so thinking that may be affecting his swimmers and not sure how this may affect the baby if we do concieve I've talked to my gynecologist about this she says this will not affect me or the baby the only effect would be for him that it can make him sterile while taking it but still very nervous. I don't fell like I could ever forgive him if anything was wrong with my baby because of that.