am I being unreasonable?

I'm 25, married, have a house and good career. I have a two year old and newborn. I refuse to go back to birth control because I just don't like it. My husband refuses a vasectomy because he's too scared. I get it. But I also do not want to get tied. So we're basically relying on condoms. Is this safe enough??  We're very good with using them correctly..