My boyfriend refuses to wear condoms

My current boyfriend won't wear condoms we have been dating for 3 years and I was on the pill so he didn't have too but my doctor suggested that I get off because it was making my body messed up. I don't understand why the women has to do the birth control why do I have to put hormones in my body and risk it messing with my body I bought the condoms and still he won't wear them he's 32 years old he said if we get pregnant it was ment to happen. I said that's crazy and just been holding out with sex