Sex without a condom

My boyfriend and I have sex pretty regularly (about 4-5 times a week) and we never use condoms. I'm on the hormonal pill, and we always use withdrawal. Anyone's opinion on if we're being safe enough?