how do you know when you're fertile?

Ma
My boyfriend and I do use condoms, but he gets mad that he can't feel much so he takes it off and pulls out ... but I get scared! Especially since I'm not sure when a woman is fertile.