Avoiding pregnancy without birth control

I absolutely hate the way my birth control has affected my body (acne, etc) plus I'm not big on messing up the normal hormonal balance of my body so I would like to get off my birth control. However, I do not want a baby at all! My partner and I already use condoms but using condoms only kind of makes me nervous... Has anyone successfully avoided pregnancy by using condoms and abstaining from sex around ovulation? Has anyone gotten pregnant accidentally from this method? Just nervous and needing advice. Thank you!