Fertile window.

Is it safe to have unprotected sex outside of your fertile window? I understand there is no 100% chance of not getting pregnant, but do we have to worry about pulling out as much outside of the fertile window?