Sex without a condom..

How worried should you be If you've had sex without a condom? Especially on your fertile days? 
(Even when you know for a fact he pulled out) 
I've always like having sex without a condom but I can't help but be paranoid whenever it does happen. It's not an always thing but some instances you just go for it.. 
I know the best way to avoid this is to obviously use condoms etc.