Avoiding pregnancy

I can't take birth control because of health reasons and I recently started having sex with my boyfriend. We use condoms and pull out every single time so I am covered when it comes to avoiding pregnancy right? Can I still have sex during my most fertile week without getting pregnant?